Skip to content

Commit 0846aca

Browse files
jjmaestrojjmaestro
committed
feat: build CI Docker images with GH Actions
Workflow to build and deploy Bazel CI Docker images This workflow builds the images, pushes them to the GHCR registry and links them with this repo. For each `Dockerfile` it builds different types of images, each corresponding with a named build stage (`... AS <NAME>`) in the `Dockerfile`. The workflow is triggered when a push to the `main`/`master` or `testing` branch contains changes to one or more of the CI `Dockerfile`s (`buildkite/docker/*/Dockerfile`). It can also be triggered manually via the Actions web UI, the GH REST API or the GH CLI tool, e.g.: ```sh gh workflow run build-ci-docker-images ``` When triggered by a `push` event the workflow will: 1. Determine which `Dockerfile`s were changed. 2. For those `Dockerfile`s, determine its build context (the `Dockerfile` directory) and the build targets (the named build stages in it). 3. Filter the build targets: * first with `RE_TARGET_INCLUDE`, which defaults to empty so it will match all the named build stages. * then with `RE_TARGET_EXCLUDE`: set by default to remove some of the build stages, e.g. the deprecated images like `centos7` and some targets that we don't want to build as images like the `nojdk` ones. 4. Then, it will also exclude all `testimage` targets because that image is only used for manually testing the workflow. 5. Finally, it will spawn a `docker/build-push-action` job for each of the build targets. For every image built, it will push to the registry three image tags: * a `sha` tag with the short hash of the commit that triggered the push * a `date` tag with the current date in ISO format (`YYYYMMDD`) * a `latest` tag When triggered manually (`workflow_dispatch` event) the workflow will default to "running in test mode": it will follow the same steps as a `push` run but with different default values (see `workflow_dispatch.inputs`): * `RE_TARGET_INCLUDE` set to `testimage` (a very simple image to exercise the build that doesn't take much compute) and * `RE_TARGET_EXCLUDE` set to the same pattern as in the `push` event. This effectively limits the build targets to only those in `testimage` not excluded by `RE_TARGET_EXCLUDE` (e.g. `nojdk`). The "test run" also limits the `PLATFORMS` to `linux/amd64`, to further reduce the cost and time of a test run. Finally, it will build those `testimage` targets but **it won't tag `latest` or push any of the image tags to the registry**. This "test mode" behavior can be changed by setting the `workflow_dispatch.inputs` variables: `RE_TARGET_EXCLUDE`, `RE_TARGET_INCLUDE`, `PLATFORMS`, `TAG_DATE`, `TAG_LATEST` and `PUSH`, e.g.: ```sh gh workflow run build-ci-docker-images \ -f RE_TARGET_INCLUDE=ubuntu2404 -f TAG_DATE=20241101 ```
1 parent 739eb39 commit 0846aca

File tree

2 files changed

+296
-0
lines changed

2 files changed

+296
-0
lines changed

‎.github/workflows/build-ci-docker-images.yaml‎

Lines changed: 294 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,294 @@
1+
# Workflow to build and deploy Bazel CI Docker images
2+
#
3+
# This workflow builds the images, pushes them to the GHCR registry and
4+
# links them with this repo.
5+
#
6+
# For each `Dockerfile` it builds different types of images, each
7+
# corresponding with a named build stage (`... AS <NAME>`) in the
8+
# `Dockerfile`.
9+
#
10+
# The workflow is triggered when a push to the `main`/`master` or
11+
# `testing` branch contains changes to one or more of the CI
12+
# `Dockerfile`s (`buildkite/docker/*/Dockerfile`). It can also be
13+
# triggered manually via the Actions web UI, the GH REST API or the GH
14+
# CLI tool, e.g.:
15+
# ```sh
16+
# gh workflow run build-ci-docker-images
17+
# ```
18+
#
19+
# When triggered by a `push` event the workflow will:
20+
#
21+
# 1. Determine which `Dockerfile`s were changed.
22+
#
23+
# 2. For those `Dockerfile`s, determine its build context (the
24+
# `Dockerfile` directory) and the build targets (the named build
25+
# stages in it).
26+
#
27+
# 3. Filter the build targets:
28+
# * first with `RE_TARGET_INCLUDE`, which defaults to empty so it will
29+
# match all the named build stages.
30+
# * then with `RE_TARGET_EXCLUDE`: set by default to remove some of
31+
# the build stages, e.g. the deprecated images like `centos7` and
32+
# some targets that we don't want to build as images like the
33+
# `nojdk` ones.
34+
#
35+
# 4. Then, it will also exclude all `testimage` targets because that
36+
# image is only used for manually testing the workflow.
37+
#
38+
# 5. Finally, it will spawn a `docker/build-push-action` job for each of
39+
# the build targets. For every image built, it will push to the
40+
# registry three image tags:
41+
# * a `sha` tag with the short hash of the commit that triggered the
42+
# push
43+
# * a `date` tag with the current date in ISO format (`YYYYMMDD`)
44+
# * a `latest` tag
45+
#
46+
# When triggered manually (`workflow_dispatch` event) the workflow will
47+
# default to "running in test mode": it will follow the same steps as a
48+
# `push` run but with different default values (see
49+
# `workflow_dispatch.inputs`):
50+
# * `RE_TARGET_INCLUDE` set to `testimage` (a very simple image to
51+
# exercise the build that doesn't take much compute) and
52+
# * `RE_TARGET_EXCLUDE` set to the same pattern as in the `push` event.
53+
#
54+
# This effectively limits the build targets to only those in `testimage`
55+
# not excluded by `RE_TARGET_EXCLUDE` (e.g. `nojdk`).
56+
#
57+
# The "test run" also limits the `PLATFORMS` to `linux/amd64`, to further
58+
# reduce the cost and time of a test run.
59+
#
60+
# Finally, it will build those `testimage` targets but **it won't tag
61+
# `latest` or push any of the image tags to the registry**.
62+
#
63+
# This "test mode" behavior can be changed by setting the
64+
# `workflow_dispatch.inputs` variables: `RE_TARGET_EXCLUDE`,
65+
# `RE_TARGET_INCLUDE`, `PLATFORMS`, `TAG_DATE`, `TAG_LATEST` and `PUSH`,
66+
# e.g.:
67+
# ```sh
68+
# gh workflow run build-ci-docker-images \
69+
# -f RE_TARGET_INCLUDE=ubuntu2404 -f TAG_DATE=20241101
70+
# ```
71+
72+
name: build-ci-docker-images
73+
74+
on:
75+
push:
76+
branches:
77+
- main
78+
- master
79+
- testing
80+
paths:
81+
# NOTE: keep in-sync with env.SPARSE_PATH
82+
- buildkite/docker/*/Dockerfile
83+
84+
workflow_dispatch:
85+
inputs:
86+
RE_TARGET_EXCLUDE:
87+
description: |-
88+
Filter out Docker targets matching this extended regex pattern
89+
# NOTE: keep in-sync with env.RE_TARGET_EXCLUDE
90+
default: ^centos7|^ubuntu16|nojdk
91+
RE_TARGET_INCLUDE:
92+
description: |-
93+
Select Docker targets matching this extended regex pattern
94+
# NOTE: keep in-sync with env.TEST_IMAGE
95+
default: testimage
96+
PLATFORMS:
97+
default: linux/amd64
98+
TAG_DATE:
99+
description: Tag image date in ISO format (YYYYMMDD)
100+
TAG_LATEST:
101+
description: Tag image version as 'latest'
102+
default: false
103+
PUSH:
104+
description: Push images to registry
105+
default: false
106+
107+
env:
108+
REGISTRY: ghcr.io
109+
SPARSE_PATH: buildkite/docker
110+
TEST_IMAGE: testimage
111+
112+
GH_EVENT_NAME: ${{ github.event_name }}
113+
GH_REF_NAME: ${{ github.ref_name }}
114+
GH_REPO: ${{ github.repository }}
115+
GH_SHA: ${{ github.sha }}
116+
GH_SHA_BEFORE: ${{ github.event.before }}
117+
118+
RE_TARGET_EXCLUDE: ${{ inputs.RE_TARGET_EXCLUDE || '^centos7|^ubuntu16|nojdk' }}
119+
RE_TARGET_INCLUDE: ${{ inputs.RE_TARGET_INCLUDE }}
120+
PLATFORMS: ${{ inputs.PLATFORMS || 'linux/amd64,linux/arm64' }}
121+
TAG_DATE: ${{ inputs.TAG_DATE }}
122+
TAG_LATEST: ${{ inputs.TAG_LATEST }}
123+
PUSH: ${{ github.event_name == 'push' || inputs.PUSH }}
124+
125+
jobs:
126+
setup-targets:
127+
runs-on: ubuntu-latest
128+
129+
defaults:
130+
run:
131+
shell: bash --noprofile --norc -euo pipefail {0}
132+
133+
outputs:
134+
targets: ${{ steps.define_targets.outputs.targets }}
135+
targets_length: ${{ steps.define_targets.outputs.targets_length }}
136+
137+
steps:
138+
- name: Sparse checkout SPARSE_PATH
139+
uses: actions/checkout@v4
140+
with:
141+
sparse-checkout-cone-mode: false
142+
sparse-checkout: |-
143+
${{ env.SPARSE_PATH }}
144+
145+
- name: Get DOCKERFILES
146+
run: |-
147+
# NOTE:
148+
# GH_SHA_BEFORE is empty on pushing the first commit of a new branch
149+
# or when running manually via workflow_dispatch
150+
if [[ -z $GH_SHA_BEFORE ]]; then
151+
DOCKERFILES="$(ls $SPARSE_PATH/*/Dockerfile)"
152+
else
153+
DOCKERFILES="$(
154+
git diff --name-only $GH_SHA_BEFORE $GH_SHA |
155+
grep Dockerfile || {
156+
# NOTE:
157+
# this grep could fail if e.g. we are force-pushing a stack
158+
# of commits where one or more commits do change Dockerfiles
159+
# but there's no change to any Dockerfile between the last
160+
# commit and this forced push.
161+
echo "WARNING: EMPTY grep" >&2
162+
true
163+
}
164+
)"
165+
fi
166+
167+
DOCKERFILES_JSON="$(echo -n "$DOCKERFILES" | jq -R '.' | jq -sc '.' )"
168+
169+
echo "DOCKERFILES_JSON=$DOCKERFILES_JSON"
170+
echo "DOCKERFILES_JSON=$DOCKERFILES_JSON" >> $GITHUB_ENV
171+
172+
- name: Define targets
173+
id: define_targets
174+
run: |-
175+
if [[ "$GH_EVENT_NAME" == "push" ]]; then
176+
RE_TARGET_EXCLUDE="$RE_TARGET_EXCLUDE|$TEST_IMAGE"
177+
fi
178+
179+
DOCKERFILES=($(echo "$DOCKERFILES_JSON" | jq -r 'join(" ")'))
180+
181+
if [[ ${#DOCKERFILES[@]} -gt 0 ]]; then
182+
TARGETS_LS="$(
183+
grep -i '^FROM .* AS ' ${DOCKERFILES[@]} |
184+
awk '{print $NF}' |
185+
{ grep -E "$RE_TARGET_INCLUDE" || true; } |
186+
{ grep -vE "$RE_TARGET_EXCLUDE" || true; } |
187+
jq -R '.'
188+
)"
189+
else
190+
TARGETS_LS=""
191+
fi
192+
193+
TARGETS="$(echo "$TARGETS_LS" | jq -sc '.')"
194+
TARGETS_LENGTH="$(echo "$TARGETS_LS" | jq -s 'length')"
195+
196+
echo "targets=$TARGETS"
197+
echo "targets_length=$TARGETS_LENGTH"
198+
199+
echo "targets=$TARGETS" >> "$GITHUB_OUTPUT"
200+
echo "targets_length=$TARGETS_LENGTH" >> "$GITHUB_OUTPUT"
201+
202+
build-and-publish-docker-images:
203+
needs: setup-targets
204+
if: ${{ needs.setup-targets.outputs.targets_length > 0 }}
205+
206+
runs-on: ubuntu-latest
207+
208+
strategy:
209+
matrix:
210+
target: ${{ fromJson(needs.setup-targets.outputs.targets) }}
211+
212+
defaults:
213+
run:
214+
shell: bash --noprofile --norc -euo pipefail {0}
215+
216+
steps:
217+
- name: Sparse checkout SPARSE_PATH
218+
uses: actions/checkout@v4
219+
with:
220+
sparse-checkout-cone-mode: false
221+
sparse-checkout: |-
222+
${{ env.SPARSE_PATH }}
223+
224+
- name: Set up dynamic env
225+
env:
226+
MATRIX_TARGET: ${{ matrix.target }}
227+
run: |-
228+
declare -A TAGS
229+
230+
TAGS[sha]="${GH_SHA::7}"
231+
232+
TAGS[date]="$TAG_DATE"
233+
# set default date value if TAG_DATE is not set, is empty
234+
# or is an empty string
235+
if [[ -z "${TAGS[date]+isset}" || -z "${TAGS[date]// }" ]]; then
236+
TAGS[date]="$(date +%Y%m%d)"
237+
fi
238+
239+
if [[ "$GH_EVENT_NAME" == "push" || "$TAG_LATEST" == "true" ]]; then
240+
TAGS[latest]="latest"
241+
fi
242+
243+
if [[ "$REGISTRY" == "gcr.io" ]]; then
244+
IMAGE_PREFIX="bazel-public"
245+
elif [[ "$REGISTRY" == "ghcr.io" ]]; then
246+
IMAGE_PREFIX="$GH_REPO"
247+
else
248+
echo "Invalid registry: $REGISTRY"
249+
exit 1
250+
fi
251+
252+
if [[ "$GH_REF_NAME" == "testing" ]]; then
253+
IMAGE_PREFIX="$IMAGE_PREFIX/testing"
254+
fi
255+
256+
IMAGE_NAME="$REGISTRY/$IMAGE_PREFIX/$MATRIX_TARGET"
257+
258+
DISTRO="${MATRIX_TARGET%%-*}"
259+
CONTEXT="$SPARSE_PATH/$DISTRO"
260+
261+
{
262+
for tag in ${!TAGS[@]}; do
263+
echo "IMAGE_TAG_${tag^^}=$IMAGE_NAME:${TAGS[$tag]}"
264+
done
265+
266+
echo "CONTEXT=$CONTEXT"
267+
} >> $GITHUB_ENV
268+
269+
- name: Set up QEMU
270+
uses: docker/setup-qemu-action@v3
271+
272+
- name: Set up Docker Buildx
273+
uses: docker/setup-buildx-action@v3
274+
275+
- name: Login to Registry
276+
uses: docker/login-action@v3
277+
with:
278+
registry: ${{ env.REGISTRY }}
279+
username: ${{ github.actor }}
280+
password: ${{ secrets.GITHUB_TOKEN }}
281+
282+
- name: Build and push
283+
uses: docker/build-push-action@v6
284+
with:
285+
context: ${{ env.CONTEXT }}
286+
target: ${{ matrix.target }}
287+
platforms: ${{ env.PLATFORMS }}
288+
tags: |-
289+
${{ env.IMAGE_TAG_SHA }}
290+
${{ env.IMAGE_TAG_DATE }}
291+
${{ env.IMAGE_TAG_LATEST }}
292+
labels: |-
293+
org.opencontainers.image.source=${{ github.repositoryUrl }}
294+
push: ${{ env.PUSH }}

‎buildkite/README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
[![build-docker-images](https://github.com/bazelbuild/continuous-integration/actions/workflows/build-docker-images.yaml/badge.svg?branch=master&event=push)](https://github.com/bazelbuild/continuous-integration/actions/workflows/build-docker-images.yaml)
2+
13
# Bazel Continuous Integration
24

35
tl;dr:

0 commit comments

Comments
 (0)