Skip to content

Add cuda_{runtim,devel} base and cuda concretization to GitHub Actions #71

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 10 commits into
base: master
Choose a base branch
from
Open

Add cuda_{runtim,devel} base and cuda concretization to GitHub Actions #71

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
d3c18be
Add cuda_runtime and cuda_devel base image builds to GitHub Actions
wdconinc Nov 26, 2025
c1ac240
Update build name to include BUILD_IMAGE
wdconinc Nov 26, 2025
b79d2fa
Clarify environment variable context in workflow
wdconinc Nov 26, 2025
209e3aa
Allow for missing arm64 digest file
wdconinc Nov 26, 2025
d93b99e
Add 'final' container target to eic matrix
wdconinc Nov 26, 2025
085e597
Change 'as' to 'AS' in Dockerfile for FromAsCasing check
wdconinc Nov 26, 2025
2f97229
Rely on transitive dependencies in GitHub Actions
wdconinc Nov 26, 2025
2ad6051
Rewrite digests step to loop over architecture artifacts dynamically …
Copilot Nov 27, 2025
bce3a3d
Improve manifest job names
wdconinc Nov 27, 2025
735cd55
Refactor eic matrix and add cuda concretization
wdconinc Nov 27, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
188 changes: 146 additions & 42 deletions .github/workflows/build-push.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,10 @@ permissions:
packages: write

env:
## CUDA version and container operating system
CUDA_VERSION: 12.5.1
CUDA_OS: ubuntu24.04

## Default versions are specified in packages.yaml but can be overridden
## note: nightly builds will always use the master/main branch
EDM4EIC_VERSION: ${{ inputs.EDM4EIC_VERSION }}
Expand All @@ -55,9 +59,26 @@ env:
INTERNAL_TAG: pipeline-${{ github.run_id }}

jobs:
env:
## The env context is not available in matrix, only in steps,
## so this job is merely to transfer env to job outputs which
## can be used in matrix.
name: Turn env into outputs
runs-on: ubuntu-latest
outputs:
CUDA_VERSION: ${{ steps.env.outputs.CUDA_VERSION }}
CUDA_OS: ${{ steps.env.outputs.CUDA_OS }}
steps:
- name: Turn env into outputs
id: env
run: |
echo "CUDA_VERSION=${{ env.CUDA_VERSION }}" >> $GITHUB_OUTPUT
echo "CUDA_OS=${{ env.CUDA_OS }}" >> $GITHUB_OUTPUT

base:
name: Build base on ${{ matrix.arch }}
name: Build ${{ matrix.BUILD_IMAGE }} on ${{ matrix.arch }}
runs-on: ${{ matrix.runner }}
needs: env
strategy:
matrix:
include:
Expand All @@ -71,6 +92,16 @@ jobs:
PLATFORM: linux/arm64
runner: ubuntu-24.04-arm
arch: arm64
- BASE_IMAGE: nvidia/cuda:${{ needs.env.outputs.CUDA_VERSION }}-devel-${{ needs.env.outputs.CUDA_OS }}
BUILD_IMAGE: cuda_devel
PLATFORM: linux/amd64
runner: ubuntu-latest
arch: amd64
- BASE_IMAGE: nvidia/cuda:${{ needs.env.outputs.CUDA_VERSION }}-runtime-${{ needs.env.outputs.CUDA_OS }}
BUILD_IMAGE: cuda_runtime
PLATFORM: linux/amd64
runner: ubuntu-latest
arch: amd64
steps:
- name: Checkout
uses: actions/checkout@v4
Expand Down Expand Up @@ -125,16 +156,16 @@ jobs:
uses: actions/cache@v4
id: cache-base-mounts
with:
path: cache-mount-base-${{ matrix.arch }}
key: ${{ matrix.arch }}-base-mounts-${{ github.ref_name }}
path: cache-mount-base-${{ matrix.BUILD_IMAGE }}-${{ matrix.arch }}
key: ${{ matrix.BUILD_IMAGE }}-${{ matrix.arch }}-base-mounts-${{ github.ref_name }}
restore-keys: |
${{ matrix.arch }}-base-mounts-main
${{ matrix.arch }}-base-mounts-
${{ matrix.BUILD_IMAGE }}-${{ matrix.arch }}-base-mounts-main
${{ matrix.BUILD_IMAGE }}-${{ matrix.arch }}-base-mounts-
- name: Inject cache mounts into builder
uses: reproducible-containers/buildkit-cache-dance@v3
with:
builder: ${{ steps.buildx.outputs.name }}
cache-dir: cache-mount-base-${{ matrix.arch }}
cache-dir: cache-mount-base-${{ matrix.BUILD_IMAGE }}-${{ matrix.arch }}
cache-map: |
{
"var-cache-apt": "/var/cache/apt",
Expand Down Expand Up @@ -195,18 +226,24 @@ jobs:
# We write this to a file for the next job
run: |
mkdir -p /tmp/digests
echo "${{ steps.meta.outputs.tags }}@${{ steps.build.outputs.digest }}" > /tmp/digests/${{ matrix.arch }}.digest
echo "${{ steps.meta.outputs.tags }}@${{ steps.build.outputs.digest }}" > /tmp/digests/${{ matrix.BUILD_IMAGE }}-${{ matrix.arch }}.digest
- name: Upload digest as artifact
uses: actions/upload-artifact@v4
with:
name: base-${{ matrix.arch }}-digest
path: /tmp/digests/${{ matrix.arch }}.digest
name: base-${{ matrix.BUILD_IMAGE }}-${{ matrix.arch }}-digest
path: /tmp/digests/${{ matrix.BUILD_IMAGE }}-${{ matrix.arch }}.digest
retention-days: 1

base-manifest:
name: Push base manifest
name: Push ${{ matrix.BUILD_IMAGE }} manifest
runs-on: ubuntu-latest
needs: base
strategy:
matrix:
include:
- BUILD_IMAGE: debian_stable_base
- BUILD_IMAGE: cuda_devel
- BUILD_IMAGE: cuda_runtime
steps:
- name: Set up QEMU (for imagetools)
uses: docker/setup-qemu-action@v3
Expand All @@ -230,30 +267,47 @@ jobs:
uses: actions/download-artifact@v4
with:
path: /tmp/digests
pattern: base-*-digest
pattern: base-${{ matrix.BUILD_IMAGE }}-*-digest
merge-multiple: true
- name: Analyze digest artifacts
id: digests
run: |
# Read the digests from the files
DIGEST_AMD64=$(cat /tmp/digests/amd64.digest)
DIGEST_ARM64=$(cat /tmp/digests/arm64.digest)
# Get the base image name from the digests (they'll be the same)
REGISTRY_IMAGE_TAG=$(echo $DIGEST_AMD64 | cut -d'@' -f1)
# Loop over all available architecture artifacts
ALL_DIGESTS=""
FIRST_DIGEST=""
shopt -s nullglob
for digest_file in /tmp/digests/${{ matrix.BUILD_IMAGE }}-*.digest; do
DIGEST=$(cat "$digest_file")
echo "Found digest: $DIGEST"
if [ -z "$ALL_DIGESTS" ]; then
ALL_DIGESTS="$DIGEST"
else
ALL_DIGESTS="$ALL_DIGESTS $DIGEST"
fi
if [ -z "$FIRST_DIGEST" ]; then
FIRST_DIGEST="$DIGEST"
fi
done
shopt -u nullglob
# Check if at least one digest was found
if [ -z "$FIRST_DIGEST" ]; then
echo "Error: No digest files found"
exit 1
fi
# Get the base image name from the first digest
REGISTRY_IMAGE_TAG=$(echo $FIRST_DIGEST | cut -d'@' -f1)
REGISTRY_IMAGE=$(echo $REGISTRY_IMAGE_TAG | cut -d':' -f1)
REGISTRY=$(echo $REGISTRY_IMAGE | cut -d'/' -f1-2)
IMAGE=$(echo $REGISTRY_IMAGE | cut -d'/' -f3)
TAG=$(echo $REGISTRY_IMAGE_TAG | cut -d':' -f2)
echo "Registry Name: $REGISTRY"
echo "Image Name: $IMAGE"
echo "Tag Name: $TAG"
echo "AMD64 Digest: $DIGEST_AMD64"
echo "ARM64 Digest: $DIGEST_ARM64"
echo "All Digests: $ALL_DIGESTS"
echo "registry=$REGISTRY" >> $GITHUB_OUTPUT
echo "image=$IMAGE" >> $GITHUB_OUTPUT
echo "tag=$TAG" >> $GITHUB_OUTPUT
echo "amd64=$DIGEST_AMD64" >> $GITHUB_OUTPUT
echo "arm64=$DIGEST_ARM64" >> $GITHUB_OUTPUT
echo "all=$ALL_DIGESTS" >> $GITHUB_OUTPUT
- name: Extract Docker metadata for final tags
id: meta
uses: docker/metadata-action@v5
Expand All @@ -275,28 +329,60 @@ jobs:
# Create the manifest list and tag it with the final tags
docker buildx imagetools create \
$TAG_ARGS \
${{ steps.digests.outputs.amd64 }} \
${{ steps.digests.outputs.arm64 }}
${{ steps.digests.outputs.all }}

eic:
name: Build ${{ matrix.BUILD_IMAGE }}${{ matrix.ENV }} on ${{ matrix.arch }}
runs-on: ${{ matrix.runner }}
needs: base-manifest
strategy:
matrix:
BUILD_IMAGE: [eic_]
BUILD_TYPE: [default]
BUILDER_IMAGE: [debian_stable_base]
RUNTIME_IMAGE: [debian_stable_base]
ENV: [ci, xl]
arch: [amd64, arm64]
include:
- arch: amd64
- BUILD_IMAGE: eic_
BUILD_TYPE: default
BUILDER_IMAGE: debian_stable_base
RUNTIME_IMAGE: debian_stable_base
ENV: ci
arch: amd64
runner: ubuntu-latest
PLATFORM: linux/amd64
target: final
- BUILD_IMAGE: eic_
BUILD_TYPE: default
BUILDER_IMAGE: debian_stable_base
RUNTIME_IMAGE: debian_stable_base
ENV: ci
arch: arm64
runner: ubuntu-24.04-arm
PLATFORM: linux/arm64
target: final
- BUILD_IMAGE: eic_
BUILD_TYPE: default
BUILDER_IMAGE: debian_stable_base
RUNTIME_IMAGE: debian_stable_base
ENV: xl
arch: amd64
runner: ubuntu-latest
PLATFORM: linux/amd64
- arch: arm64
target: final
- BUILD_IMAGE: eic_
BUILD_TYPE: default
BUILDER_IMAGE: debian_stable_base
RUNTIME_IMAGE: debian_stable_base
ENV: xl
arch: arm64
runner: ubuntu-24.04-arm
PLATFORM: linux/arm64
target: final
- BUILD_IMAGE: eic_
BUILD_TYPE: default
BUILDER_IMAGE: cuda_devel
RUNTIME_IMAGE: cuda_devel
ENV: cuda
arch: amd64
runner: ubuntu-latest
PLATFORM: linux/amd64
target: builder_concretization_default
steps:
- name: Free Disk Space (Ubuntu)
uses: jlumbroso/free-disk-space@v1.3.1
Expand Down Expand Up @@ -401,6 +487,7 @@ jobs:
secret-files: |
mirrors=mirrors.yaml
platforms: ${{ matrix.PLATFORM }}
target: ${{ matrix.target }}
labels: ${{ steps.meta.outputs.labels }}
outputs: type=image,name=${{ env.GH_REGISTRY }}/${{ env.GH_REGISTRY_USER }}/${{ matrix.BUILD_IMAGE }}${{ matrix.ENV }},push-by-digest=true,name-canonical=true,push=true
build-args: |
Expand Down Expand Up @@ -434,11 +521,12 @@ jobs:
retention-days: 1

eic-manifest:
name: Push eic manifest
name: Push ${{ matrix.BUILD_IMAGE }}${{ matrix.ENV }} manifest
runs-on: ubuntu-latest
needs: eic
strategy:
matrix:
BUILD_IMAGE: [eic_]
ENV: [ci, xl]
steps:
- name: Set up QEMU (for imagetools)
Expand Down Expand Up @@ -468,25 +556,42 @@ jobs:
- name: Analyze digest artifacts
id: digests
run: |
# Read the digests from the files
DIGEST_AMD64=$(cat /tmp/digests/amd64.digest)
DIGEST_ARM64=$(cat /tmp/digests/arm64.digest)
# Get the base image name from the digests (they'll be the same)
REGISTRY_IMAGE_TAG=$(echo $DIGEST_AMD64 | cut -d'@' -f1)
# Loop over all available architecture artifacts
ALL_DIGESTS=""
FIRST_DIGEST=""
shopt -s nullglob
for digest_file in /tmp/digests/*.digest; do
DIGEST=$(cat "$digest_file")
echo "Found digest: $DIGEST"
if [ -z "$ALL_DIGESTS" ]; then
ALL_DIGESTS="$DIGEST"
else
ALL_DIGESTS="$ALL_DIGESTS $DIGEST"
fi
if [ -z "$FIRST_DIGEST" ]; then
FIRST_DIGEST="$DIGEST"
fi
done
shopt -u nullglob
# Check if at least one digest was found
if [ -z "$FIRST_DIGEST" ]; then
echo "Error: No digest files found"
exit 1
fi
# Get the base image name from the first digest
REGISTRY_IMAGE_TAG=$(echo $FIRST_DIGEST | cut -d'@' -f1)
REGISTRY_IMAGE=$(echo $REGISTRY_IMAGE_TAG | cut -d':' -f1)
REGISTRY=$(echo $REGISTRY_IMAGE | cut -d'/' -f1-2)
IMAGE=$(echo $REGISTRY_IMAGE | cut -d'/' -f3)
TAG=$(echo $REGISTRY_IMAGE_TAG | cut -d':' -f2)
echo "Registry Name: $REGISTRY"
echo "Image Name: $IMAGE"
echo "Tag Name: $TAG"
echo "AMD64 Digest: $DIGEST_AMD64"
echo "ARM64 Digest: $DIGEST_ARM64"
echo "All Digests: $ALL_DIGESTS"
echo "registry=$REGISTRY" >> $GITHUB_OUTPUT
echo "image=$IMAGE" >> $GITHUB_OUTPUT
echo "tag=$TAG" >> $GITHUB_OUTPUT
echo "amd64=$DIGEST_AMD64" >> $GITHUB_OUTPUT
echo "arm64=$DIGEST_ARM64" >> $GITHUB_OUTPUT
echo "all=$ALL_DIGESTS" >> $GITHUB_OUTPUT
- name: Extract Docker metadata for final tags
id: meta
uses: docker/metadata-action@v5
Expand All @@ -508,5 +613,4 @@ jobs:
# Create the manifest list and tag it with the final tags
docker buildx imagetools create \
$TAG_ARGS \
${{ steps.digests.outputs.amd64 }} \
${{ steps.digests.outputs.arm64 }}
${{ steps.digests.outputs.all }}
2 changes: 1 addition & 1 deletion containers/eic/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -286,7 +286,7 @@ EOF
## ========================================================================================
## final image, based on runtime_installation_custom
## ========================================================================================
FROM runtime_installation_custom
FROM runtime_installation_custom AS final
ARG TARGETPLATFORM

# Open Container Initiative labels
Expand Down
Loading