[CDRIVER-6142] Refactor dependency/environment setup in Earthfile #2165

vector-of-bool · 2025-10-30T21:46:33Z

Summary

Refer: CDRIVER-6142, this is a set of drive-by changes done as part of CDRIVER-5982, split into a separate PR to keep a clean history and review set.

This changeset modifies how Earthly environments are specified and set up for the build.

Most of this changeset consists of modifications to generated Evergreen configs.

While doing refactors to the amongoc CI plan, I found that it is far simpler to have an Earthly function for each dependency that knows how to "do the right thing" to install the relevant packages for whatever host platform it happens to be running within, especially by adding a new __can_install utility. The following significant changes are made:

The env.foo Earthly targets are dropped. Instead, environments are specified using --from instead of --env, which just executes a regular FROM command to set the base container image.
For each dependency, an ADD_FOO Earthly function is defined, and it will just "do the right thing" to obtain the relevant package for the build (or do nothing, if that dependency is disabled).
The many small scripts are consolidated into a single __tool script, which explodes itself using __alias to reproduce the same set of tools that were available beforehand.
New tools are defined, the most important being __can_install <pkg>, which determines whether a package <pkg> is available for installation. This makes dep management far simpler since we don't need to do fancy platform/tool detection when we can just say "If gcc-c++ is available, install it." without needing to know which package manager we're going to use.
Because --from (formerly --env) has been modified to take a plain container image specifier, the logic for switching to ECR for the base platform container images is moved to earthly.py.
The build has been split into four steps: +init, +build-environment, +configure, and +build. The main purpose of this split is to support subdividing the EVG build into the four steps so that we get separate timing/logging information for each.

All of the above also has the added benefit of being able to test any Linux distro/version on-the-fly by just passing a new --from rather than needing to modify Earthfile to support a new platform.

This change rewrites much of Earthfile's dependency-installation handling code to be simpler to modify, and more flexible to use. Previously, each environment had its own target that needed to know how to install all of our dependencies for the specific platform. This refactor inverts the relationship: Instead, each dependency has an Earthly function that knows how to install itself based on whatever platform it happens to be running within. This allows the FROM line to be a plain image, rather that one of our handcrafted `env.foo` targets. This will make introducing and testing additional environments far more simple in the future. This change also allows us to remove the tools/build.earth utility, since we can now fit everything in the top-level file.

vector-of-bool · 2025-11-10T16:26:29Z

@eramongodb requested you especially because this does some juggling of the ECR handling in EVG.

eramongodb

The Earthly workflow is a lot easier to track than before. Thank you for these improvements!

eramongodb · 2025-11-10T18:58:07Z

.evergreen/config_generator/components/earthly.py

        # Match 'alpine3.18' 'alpine53.123' etc.
-        case alp if mat := re.match(r'alpine(\d+\.\d+)', alp):
+        case alp if mat := re.match(r'alpine:(\d+\.\d+)', alp):
            return ('Alpine', mat[1])
        case 'archlinux':
            return 'ArchLinux', None
        # Match 'u22', 'u20', 'u71' etc.
-        case ubu if mat := re.match(r'u(\d\d)', ubu):
-            return 'Ubuntu', f'{mat[1]}.04'
+        case ubu if mat := re.match(r'ubuntu:(\d\d.*)', ubu):
+            return 'Ubuntu', f'{mat[1]}'
        # Match 'centos9', 'centos10', etc.
-        case cent if mat := re.match(r'centos(\d+)', cent):
-            return 'CentOS', f'{mat[1]}'
+        case cent if mat := re.match(r'.*centos:(stream)?(\d+.*)', cent):
+            return 'CentOS', f'{mat[2]}'
        # Match 'almalinux8', 'almalinux10', etc.
-        case alm if mat := re.match(r'almalinux(\d+)', alm):
+        case alm if mat := re.match(r'almalinux:(\d+.*)', alm):
            return 'AlmaLinux', f'{mat[1]}'


Comments need to be updated for consistency with new patterns.

eramongodb · 2025-11-10T19:05:00Z

.evergreen/config_generator/components/earthly.py

+        case cent if mat := re.match(r'.*centos:(stream)?(\d+.*)', cent):
+            return 'CentOS', f'{mat[2]}'


Suggested change

case cent if mat := re.match(r'.*centos:(stream)?(\d+.*)', cent):

return 'CentOS', f'{mat[2]}'

case cent if mat := re.match(r'.*centos:(?:stream)?(\d+.*)', cent):

return 'CentOS', f'{mat[1]}'

Use (?:<...>) to match <...> without including it in resulting match groups.

Can the (\d+.*) be changed to (\d+)? I don't think we intend to use any -blah images (minimal, dev, date, etc.).

eramongodb · 2025-11-10T19:23:54Z

.evergreen/config_generator/components/earthly.py

+def from_container_image(img: EnvImage) -> str:
+    """
+    Modify an unqualified FROM container identifier to route to our ECR host
+    """
+    if '/' in img or img.startswith('+'):
+        return img
+    return f'{_ECR_HOST}/dockerhub/library/{img}'


Suggest including a reference to DEVPROD-21478 for discoverability once Amazon ECR registry-handling is implemented in EVG (specifically RHEL distros) and we no longer have to workaround it ourselves.

eramongodb · 2025-11-10T20:06:12Z

.evergreen/config_generator/components/earthly.py

                args={'targets': ' '.join(targets)} | earthly_args,
            ),
        ],  # type: ignore (The type annots on `commands` is wrong)
        tags=['earthly', 'pr-merge-gate', *env_tags],


This is a pre-existing issue (not a regression in this PR0, but the tag list for Earthly tasks could use some cleanup, e.g. tasks in the archlinux-clang variant include all OS image tags:

Suggest taking this opportunity to restrict the OS image tags to the corresponding build variant: e.g. I expect tasks in the archlinux-clang to have only the following tags at most:

earthly

pr-merge-gate

archlinux-clang

eramongodb · 2025-11-10T20:19:19Z

tools/__tool

@@ -0,0 +1,311 @@
+#!/usr/bin/env sh


Document that this file/script is intended for use via Earthly only (notably, includes modifications to system environment and system paths).

eramongodb · 2025-11-10T21:06:52Z

tools/__tool

+    elif test -f /etc/SuSE-brand \
+        || (test -f /etc/os-release && grep "opensuse" /etc/os-release); then
+        test $# = 0 || zypper --non-interactive install "$@" || return


Is this necessary? AFAIK we have never tested with OpenSUSE before. Is adding OpenSUSE coverage planned?

eramongodb · 2025-11-10T21:07:23Z

tools/__tool

+        if test -f /usr/bin/dnf; then
+            # 'dnf' will "do the right thing"
+            test $# = 0 || dnf install -y "$@" || return


Do we need dnf when we always have yum in our RHEL-like distros?

eramongodb · 2025-11-10T21:07:52Z

tools/__tool

+        # a TTY.
+        test $# = 0 || env DEBIAN_FRONTEND=noninteractive \
+            apt-get -y install -- "$@" || return
+    elif test -f /etc/redhat-release || grep 'ID="amzn"' /etc/os-release >/dev/null 1>&2; then


Is ID="amzn" testing for Amazon Linux? Do we plan to extend Earthly coverage to Amazon Linux? (Note: EVG distros for this OS are available and are not limited resources.)

eramongodb · 2025-11-10T21:11:09Z

Earthfile

+    # Run the add-ccache command here. This needs to run directly within the same
+    # target that makes use of it, to ensure that the CACHE line has an effect
+    # within that target
+    DO --pass-args +ADD_CCACHE


Is there a point to adding ccache support within Earthly tasks? AFAIK the ccache cache directory will be isolated within each temporary container it is executed within, then immediately thrown away along with the temporary container. There is no opportunity for reuse unless an existing cache directory is copied/preserved across Earthly tasks, such as (I think) via --mount type=cache.

eramongodb · 2025-11-10T21:20:53Z

Earthfile

+    ARG lld = on
+    IF __bool "$lld"
+        IF __can_install lld
+            RUN __do "Installing LLD linker..." __install lld


Suggest adding a reference to CDRIVER-6150 to help track required changes should MONGO_USE_LLD be deprecated/removed.

vector-of-bool added 13 commits October 30, 2025 11:00

Consolidate several small scripts into a single __tool

a035401

Update Earthly-generated tasks

7b73e94

Simplify some path handling in Earthfile

c22a137

Fix: Allow Ccache to cache properly in Earthly targets

f488eab

Split out an +init target

358a337

Update Earthly target documentation

ec044b1

Rearrange some Earthly documentation

e5d322f

Fix: Wrong build paths to +verify-headers

31e88f8

Add Ubuntu 24.04

bdd5741

Enable building both with/without Snappy in Earthly

71ffffc

Tasks with/without Snappy

1f7d32a

Merge branch 'master' into CDRIVER-6142-earthfile-refactor

082ca14

vector-of-bool marked this pull request as ready for review November 10, 2025 16:25

vector-of-bool requested a review from a team as a code owner November 10, 2025 16:25

vector-of-bool requested review from connorsmacd and eramongodb and removed request for connorsmacd November 10, 2025 16:25

eramongodb requested changes Nov 10, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[CDRIVER-6142] Refactor dependency/environment setup in Earthfile #2165

[CDRIVER-6142] Refactor dependency/environment setup in Earthfile #2165

vector-of-bool commented Oct 30, 2025

Uh oh!

vector-of-bool commented Nov 10, 2025

Uh oh!

eramongodb left a comment

Uh oh!

eramongodb Nov 10, 2025

Uh oh!

eramongodb Nov 10, 2025

Uh oh!

eramongodb Nov 10, 2025

Uh oh!

eramongodb Nov 10, 2025

Uh oh!

eramongodb Nov 10, 2025

Uh oh!

eramongodb Nov 10, 2025

Uh oh!

eramongodb Nov 10, 2025

Uh oh!

eramongodb Nov 10, 2025

Uh oh!

eramongodb Nov 10, 2025

Uh oh!

eramongodb Nov 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		case cent if mat := re.match(r'.centos:(stream)?(\d+.)', cent):
		return 'CentOS', f'{mat[2]}'

[CDRIVER-6142] Refactor dependency/environment setup in Earthfile #2165

Are you sure you want to change the base?

[CDRIVER-6142] Refactor dependency/environment setup in Earthfile #2165

Conversation

vector-of-bool commented Oct 30, 2025

Summary

Uh oh!

vector-of-bool commented Nov 10, 2025

Uh oh!

eramongodb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants