Make extensions reproducible, add lock files (#1785)

Returns `module_ctx.extension_metadata(reproducible = True)` from
module extensions and checks in `MODULE.bazel.lock` files. Adds
`--lockfile_mode=error` to the top level `.bazelrc`, and adds
`--lockfile_mode=update` where necessary to ensure tests don't break.

The following command reveals the substantial part of this change:

```txt
$ git diff --stat HEAD^ ':!:**MODULE.bazel.lock'
 .bazelci/presubmit.yml                                |  1 +
 .bazelrc                                              |  6 ++++++
 .gitignore                                            |  7 ++-----
 dt_patches/test_dt_patches/.bazelrc                   |  5 +++++
 dt_patches/test_dt_patches_user_srcjar/.bazelrc       |  6 ++++++
 dt_patches/test_dt_patches_user_srcjar/extensions.bzl |  4 +++-
 examples/crossbuild/.bazelrc                          | 12 ++++++++++++
 scala/extensions/config.bzl                           |  1 +
 scala/extensions/deps.bzl                             | 22 ++++++++++++++++++----
 scala/extensions/protoc.bzl                           |  3 +++
 scala/private/extensions/dev_deps.bzl                 |  1 +
 test/compiler_sources_integrity/.bazelrc              |  5 +++++
 test/shell/test_bzlmod_macros.sh                      |  4 +++-
 test_version.sh                                       |  4 +++-
 14 files changed, 69 insertions(+), 12 deletions(-)
```

This change commits the top level `MODULE.bazel.lock` file and the lock
files from all nested modules where tests run. It excludes lock files
from modules that are consumed by other test modules, but where tests
don't actually run, specifically:

- `deps/latest`
- `dt_patches/compiler_sources`

The test module lock files reflect the state after `./test_all.sh`
completes successfully. Some test modules explicitly set
`--lockfile_mode=update`, so their lock file contents may change during
test runs.

---

At @jayconrod's suggestion, it seems worth revisiting the
`MODULE.bazel.lock` mechanism. The format appears to have stabilized
(especially in newer Bazels), and it seems explicitly marking extensions
as reproducible helps shrink lock files dramatically. While I've yet to
notice (or measure) performance benefits, the stability and security
benefits appear worth the effort.

The following comments explain different aspects of this change in
greater detail.

---

The concept of module extension "reproducibility" helps avoid
unnecessary `MODULE.bazel.lock` updates. This why most extensions now
return `extension_metadata` with `reproducibility = True`:

- https://bazel.build/versions/7.6.0/external/extension#specify_reproducibility

Before marking `scala_deps` as reproducible, the `MODULE.bazel.lock`
file was 30892 lines long. This dramatic effect on our own lock file
suggests that this change will potentially help consumers maintain
smaller lock files as well.

---

Setting `common --lockfile_mode=error` in `.bazelrc` ensures Bazel
raises an error when it detects any `MODULE.bazel.lock` files are out of
date. However, there are situations where we need to relax this strict
setting:

- Updating `MODULE.bazel.lock` files after editing `MODULE.bazel` files
  or module extensions.
- Running tests whose modules depend upon nonreproducible extensions.
- Running tests that generate their own temporary test modules.
- Running the entire test suite with different versions of Bazel.

The rest of this commit message describes how this change handles each
of these cases.

---

Per the new `.bazelrc` comment, we need to set `--lockfile_mode=refresh`
when actually intend to update any `MODULE.bazel.lock` files.

---

The following test modules depend upon nonreproducible extensions, which
require `MODULE.bazel.lock` updates during test runs:

1. `dt_patches/test_dt_patches`
2. `dt_patches/test_dt_patches_user_srcjar`
3. `test/compiler_sources_integrity`

Specifically:

- Modules 1. and 2. use the `compiler_sources_repo` extension from
  `dt_patches/compiler_sources`, which instantiates Maven artifact repos
  without supplying any integrity values. This is the textbook
  definition of a nonreproducible module extension.

- Modules 2. and 3. contain `scala_deps.compiler_srcjar` tags without a
  `label`, `sha256`, or `integrity` attribute. This should prove so
  limited a case as to be nonexistent in practice. Even so, the
  `scala_deps` extension makes sure to indicate that it's _not_
  reproducible in that case.

  Diffing the `dt_patches` test module lock files shows what happens
  when a `compiler_srcjar` tag lacks those attributes:

  ```sh
  diff dt_patches/test_dt_patches{,_user_srcjar}/MODULE.bazel.lock
  ```

These modules' `.bazelrc` files now set `common --lockfile_mode=update`
after importing the top level `.bazelrc`. This prevents Bazel breakages
by overriding the inherited `common --lockfile_mode=error` setting.

---

Tests that generate their own test modules fail under
`--lockfile_mode=error` because Bazel needs to generate
`MODULE.bazel.lock` files for such modules. The following test
scripts, which previously only copied the top level `.bazelrc` into
their generated modules, now use `sed` to set `--lockfile_mode=update`:

- `test/shell/test_bzlmod_macros.sh`
- `test_version.sh`

---

Finally, running tests with different versions of Bazel while
`--lockfile_mode=error` is in effect will fail. This is because the lock
file version generated by a specific Bazel version isn't compatible with
any other Bazel version.

- The `.bazelrc` line introducing the `--lockfile_mode` flag has a
  comment suggesting that users set it to `update` when testing other
  Bazel versions.

- `.bazelci/presubmit.yml` now applies `--lockfile_mode=update` to the
  `last_green` job.

- `examples/crossbuild/.bazelrc` now applies `--lockfile_mode=update`.
  This ensures that the Bazel Central Registry presubmit jobs based on
  this module, which run under a matrix of different Bazel versions,
  will all succeed.

  Otherwise, the `bazel --nosystem_rc --nohome_rc version` setup command
  will break before the `build_flags` and `test_flags` attributes apply.
  (Note this command _doesn't_ use `--noworkspace_rc`.)

Author: mbland

.bazelci/presubmit.yml

@@ -41,6 +41,7 @@ tasks:
     shell_commands:
     # Install xmllint
     - sudo apt update && sudo apt install --reinstall libxml2-utils -y
+    - echo "common --lockfile_mode=update" >>.bazelrc
     - "./test_rules_scala.sh"
     soft_fail:
       - exit_status: "*"

.bazelrc

@@ -1,6 +1,12 @@
 # Remove once Bazel 8 becomes the minimum supported version.
 common --noenable_workspace --incompatible_use_plus_in_repo_names
 
+# Ensure that the `MODULE.bazel.lock` files don't change silently.
+# - Set to `refresh` to actually update the lockfiles.
+# - Set to `update` when testing a Bazel version different from .bazelversion
+#   (or comment it out, since `update` is the default).
+common --lockfile_mode=error
+
 # Uncomment to run tests under `WORKSPACE`. Remove once Bazel 9 becomes the
 # minimum supported version.
 #common --enable_workspace --noenable_bzlmod

.gitignore

@@ -20,12 +20,9 @@ repository-artifacts.json
 # From scripts/*.py
 **/__pycache__/
 
-# Until it settles down
-**/MODULE.bazel.lock
-
 # Used by some tests, but can also be used for local experimentation.
 tmp/
 
 # Not required by tests.
-deps/latest/.bazelversion
-
+deps/latest/MODULE.bazel.lock
+dt_patches/compiler_sources/MODULE.bazel.lock

MODULE.bazel.lock

@@ -1 +1,6 @@
 import ../../.bazelrc
+
+# `MODULE.bazel` uses the nonreproducible `compiler_source_repos` module
+# extension. As a result, `--repo_env=SCALA_VERSION` values from test cases
+# change the `MODULE.bazel.lock` file.
+common --lockfile_mode=update

dt_patches/test_dt_patches/.bazelrc

@@ -1 +1,7 @@
 import ../../.bazelrc
+
+# `MODULE.bazel` uses the nonreproducible `compiler_source_repos` module
+# extension and contains `scala_deps.compiler_srcjar` tags without `sha256`,
+# `integrity`, or `label` attributes. As a result, `--repo_env=SCALA_VERSION`
+# values from test cases change the `MODULE.bazel.lock` file.
+common --lockfile_mode=update

dt_patches/test_dt_patches/MODULE.bazel.lock

@@ -13,8 +13,10 @@ def import_compiler_user_srcjar_repos():
         url = "https://repo1.maven.org/maven2/org/scala-lang/scala3-compiler_3/3.4.3/scala3-compiler_3-3.4.3-sources.jar",
     )
 
-def _compiler_user_srcjar_repos_impl(_ctx):
+def _compiler_user_srcjar_repos_impl(module_ctx):
     import_compiler_user_srcjar_repos()
+    return module_ctx.extension_metadata(reproducible = True)
+
 
 compiler_user_srcjar_repos = module_extension(
     implementation = _compiler_user_srcjar_repos_impl,

dt_patches/test_dt_patches_user_srcjar/.bazelrc

@@ -1 +1,13 @@
 import ../../.bazelrc
+
+# The Bazel Central Registry presubmit jobs in `.bazelci/presubmit.yml` and
+# `.bcr/presubmit.yml` use this module. However, those jobs run `bazel
+# --nosystem_rc --nohome_rc version` before the `build_flags` and `test_flags`
+# attributes apply. (Note this command _doesn't_ use `--noworkspace_rc`.)
+#
+# Without the following override, `--lockfile_mode=error` still applies, even if
+# `--lockfile_mode=update` appears in `build_flags` and `test_flags`. Since
+# these jobs run under a matrix of different Bazel versions, most jobs will
+# break without this flag. This is because the lock file version generated by a
+# specific Bazel version isn't compatible with any other Bazel version.
+common --lockfile_mode=update