Skip to content

Commit 713e477

Browse files
MoeMahhoukmichaelsproul
MoeMahhouk
and
michaelsproul
authored
feat: Add reproducible builds release workflows and push images to DockerHub (#7614)
This pull request introduces workflows and updates to ensure reproducible builds for the Lighthouse project. It adds two GitHub Actions workflows for building and testing reproducible Docker images and binaries, updates the `Makefile` to streamline reproducible build configurations, and modifies the `Dockerfile.reproducible` to align with the new build process. Additionally, it removes the `reproducible` profile from `Cargo.toml`. ### New GitHub Actions Workflows: * [`.github/workflows/docker-reproducible.yml`](diffhunk://#diff-222af23bee616920b04f5b92a83eb5106fce08abd885cd3a3b15b8beb5e789c3R1-R145): Adds a workflow to build and push reproducible multi-architecture Docker images for releases, including support for dry runs without pushing an image. ### Build Configuration Updates: * [`Makefile`](diffhunk://#diff-76ed074a9305c04054cdebb9e9aad2d818052b07091de1f20cad0bbac34ffb52L85-R143): Refactors reproducible build targets, centralizes environment variables for reproducibility, and updates Docker build arguments for `x86_64` and `aarch64` architectures. * [`Dockerfile.reproducible`](diffhunk://#diff-587298ff141278ce3be7c54a559f9f31472cc5b384e285e2105b3dee319ba31dL1-R24): Updates the base Rust image to version 1.86, removes hardcoded reproducibility settings, and delegates build logic to the `Makefile`. * Switch to using jemalloc-sys from Debian repos instead of building it from source. A Debian version is [reproducible](https://tests.reproducible-builds.org/debian/rb-pkg/trixie/amd64/jemalloc.html) which is [hard to achieve](NixOS/nixpkgs#380852) if you build it from source. ### Profile Removal: * [`Cargo.toml`](diffhunk://#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542L289-L295): Removes the `reproducible` profile, simplifying build configurations and relying on external tooling for reproducibility. Co-Authored-By: Moe Mahhouk <mohammed-mahhouk@hotmail.com> Co-Authored-By: chonghe <44791194+chong-he@users.noreply.github.com> Co-Authored-By: Michael Sproul <michaelsproul@users.noreply.github.com>
1 parent 847fa3f commit 713e477

File tree

6 files changed

+231
-49
lines changed

6 files changed

+231
-49
lines changed

‎.github/workflows/docker-reproducible.yml‎

Lines changed: 176 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,176 @@
1+
name: docker-reproducible
2+
3+
on:
4+
push:
5+
branches:
6+
- unstable
7+
- stable
8+
tags:
9+
- v*
10+
workflow_dispatch: # allows manual triggering for testing purposes and skips publishing an image
11+
12+
env:
13+
DOCKER_REPRODUCIBLE_IMAGE_NAME: >-
14+
${{ github.repository_owner }}/lighthouse-reproducible
15+
DOCKER_PASSWORD: ${{ secrets.DH_KEY }}
16+
DOCKER_USERNAME: ${{ secrets.DH_ORG }}
17+
18+
jobs:
19+
extract-version:
20+
name: extract version
21+
runs-on: ubuntu-22.04
22+
steps:
23+
- name: Extract version
24+
run: |
25+
if [[ "${{ github.ref }}" == refs/tags/* ]]; then
26+
# It's a tag (e.g., v1.2.3)
27+
VERSION="${GITHUB_REF#refs/tags/}"
28+
elif [[ "${{ github.ref }}" == refs/heads/stable ]]; then
29+
# stable branch -> latest
30+
VERSION="latest"
31+
elif [[ "${{ github.ref }}" == refs/heads/unstable ]]; then
32+
# unstable branch -> latest-unstable
33+
VERSION="latest-unstable"
34+
else
35+
# For manual triggers from other branches and will not publish any image
36+
VERSION="test-build"
37+
fi
38+
echo "VERSION=$VERSION" >> $GITHUB_OUTPUT
39+
id: extract_version
40+
outputs:
41+
VERSION: ${{ steps.extract_version.outputs.VERSION }}
42+
43+
verify-and-build:
44+
name: verify reproducibility and build
45+
needs: extract-version
46+
strategy:
47+
matrix:
48+
arch: [amd64, arm64]
49+
include:
50+
- arch: amd64
51+
rust_target: x86_64-unknown-linux-gnu
52+
rust_image: >-
53+
rust:1.88-bullseye@sha256:8e3c421122bf4cd3b2a866af41a4dd52d87ad9e315fd2cb5100e87a7187a9816
54+
platform: linux/amd64
55+
runner: ubuntu-22.04
56+
- arch: arm64
57+
rust_target: aarch64-unknown-linux-gnu
58+
rust_image: >-
59+
rust:1.88-bullseye@sha256:8b22455a7ce2adb1355067638284ee99d21cc516fab63a96c4514beaf370aa94
60+
platform: linux/arm64
61+
runner: ubuntu-22.04-arm
62+
runs-on: ${{ matrix.runner }}
63+
steps:
64+
- uses: actions/checkout@v4
65+
66+
- name: Set up Docker Buildx
67+
uses: docker/setup-buildx-action@v3
68+
with:
69+
driver: docker
70+
71+
- name: Verify reproducible builds (${{ matrix.arch }})
72+
run: |
73+
# Build first image
74+
docker build -f Dockerfile.reproducible \
75+
--platform ${{ matrix.platform }} \
76+
--build-arg RUST_TARGET="${{ matrix.rust_target }}" \
77+
--build-arg RUST_IMAGE="${{ matrix.rust_image }}" \
78+
-t lighthouse-verify-1-${{ matrix.arch }} .
79+
80+
# Extract binary from first build
81+
docker create --name extract-1-${{ matrix.arch }} lighthouse-verify-1-${{ matrix.arch }}
82+
docker cp extract-1-${{ matrix.arch }}:/lighthouse ./lighthouse-1-${{ matrix.arch }}
83+
docker rm extract-1-${{ matrix.arch }}
84+
85+
# Clean state for second build
86+
docker buildx prune -f
87+
docker system prune -f
88+
89+
# Build second image
90+
docker build -f Dockerfile.reproducible \
91+
--platform ${{ matrix.platform }} \
92+
--build-arg RUST_TARGET="${{ matrix.rust_target }}" \
93+
--build-arg RUST_IMAGE="${{ matrix.rust_image }}" \
94+
-t lighthouse-verify-2-${{ matrix.arch }} .
95+
96+
# Extract binary from second build
97+
docker create --name extract-2-${{ matrix.arch }} lighthouse-verify-2-${{ matrix.arch }}
98+
docker cp extract-2-${{ matrix.arch }}:/lighthouse ./lighthouse-2-${{ matrix.arch }}
99+
docker rm extract-2-${{ matrix.arch }}
100+
101+
# Compare binaries
102+
echo "=== Comparing binaries ==="
103+
echo "Build 1 SHA256: $(sha256sum lighthouse-1-${{ matrix.arch }})"
104+
echo "Build 2 SHA256: $(sha256sum lighthouse-2-${{ matrix.arch }})"
105+
106+
if cmp lighthouse-1-${{ matrix.arch }} lighthouse-2-${{ matrix.arch }}; then
107+
echo "Reproducible build verified for ${{ matrix.arch }}"
108+
else
109+
echo "Reproducible build FAILED for ${{ matrix.arch }}"
110+
echo "BLOCKING RELEASE: Builds are not reproducible!"
111+
echo "First 10 differences:"
112+
cmp -l lighthouse-1-${{ matrix.arch }} lighthouse-2-${{ matrix.arch }} | head -10
113+
exit 1
114+
fi
115+
116+
# Clean up verification artifacts but keep one image for publishing
117+
rm -f lighthouse-*-${{ matrix.arch }}
118+
docker rmi lighthouse-verify-1-${{ matrix.arch }} || true
119+
120+
# Re-tag the second image for publishing (we verified it's identical to first)
121+
VERSION=${{ needs.extract-version.outputs.VERSION }}
122+
FINAL_TAG="${{ env.DOCKER_REPRODUCIBLE_IMAGE_NAME }}:${VERSION}-${{ matrix.arch }}"
123+
docker tag lighthouse-verify-2-${{ matrix.arch }} "$FINAL_TAG"
124+
125+
- name: Log in to Docker Hub
126+
if: ${{ github.event_name != 'workflow_dispatch' }}
127+
uses: docker/login-action@v3
128+
with:
129+
username: ${{ env.DOCKER_USERNAME }}
130+
password: ${{ env.DOCKER_PASSWORD }}
131+
132+
- name: Push verified image (${{ matrix.arch }})
133+
if: ${{ github.event_name != 'workflow_dispatch' }}
134+
run: |
135+
VERSION=${{ needs.extract-version.outputs.VERSION }}
136+
IMAGE_TAG="${{ env.DOCKER_REPRODUCIBLE_IMAGE_NAME }}:${VERSION}-${{ matrix.arch }}"
137+
docker push "$IMAGE_TAG"
138+
139+
- name: Clean up local images
140+
run: |
141+
docker rmi lighthouse-verify-2-${{ matrix.arch }} || true
142+
VERSION=${{ needs.extract-version.outputs.VERSION }}
143+
docker rmi "${{ env.DOCKER_REPRODUCIBLE_IMAGE_NAME }}:${VERSION}-${{ matrix.arch }}" || true
144+
145+
- name: Upload verification artifacts (on failure)
146+
if: failure()
147+
uses: actions/upload-artifact@v4
148+
with:
149+
name: verification-failure-${{ matrix.arch }}
150+
path: |
151+
lighthouse-*-${{ matrix.arch }}
152+
153+
create-manifest:
154+
name: create multi-arch manifest
155+
runs-on: ubuntu-22.04
156+
needs: [extract-version, verify-and-build]
157+
if: ${{ github.event_name != 'workflow_dispatch' }}
158+
steps:
159+
- name: Log in to Docker Hub
160+
uses: docker/login-action@v3
161+
with:
162+
username: ${{ env.DOCKER_USERNAME }}
163+
password: ${{ env.DOCKER_PASSWORD }}
164+
165+
- name: Create and push multi-arch manifest
166+
run: |
167+
IMAGE_NAME=${{ env.DOCKER_REPRODUCIBLE_IMAGE_NAME }}
168+
VERSION=${{ needs.extract-version.outputs.VERSION }}
169+
170+
# Create manifest for the version tag
171+
docker manifest create \
172+
${IMAGE_NAME}:${VERSION} \
173+
${IMAGE_NAME}:${VERSION}-amd64 \
174+
${IMAGE_NAME}:${VERSION}-arm64
175+
176+
docker manifest push ${IMAGE_NAME}:${VERSION}

‎Cargo.toml‎

Lines changed: 0 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -279,13 +279,6 @@ lto = "fat"
279279
codegen-units = 1
280280
incremental = false
281281

282-
[profile.reproducible]
283-
inherits = "release"
284-
debug = false
285-
panic = "abort"
286-
codegen-units = 1
287-
overflow-checks = true
288-
289282
[profile.release-debug]
290283
inherits = "release"
291284
debug = true

‎Dockerfile.reproducible‎

Lines changed: 6 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -3,42 +3,22 @@ ARG RUST_IMAGE="rust:1.88-bullseye@sha256:8e3c421122bf4cd3b2a866af41a4dd52d87ad9
33
FROM ${RUST_IMAGE} AS builder
44

55
# Install specific version of the build dependencies
6-
RUN apt-get update && apt-get install -y libclang-dev=1:11.0-51+nmu5 cmake=3.18.4-2+deb11u1
6+
RUN apt-get update && apt-get install -y libclang-dev=1:11.0-51+nmu5 cmake=3.18.4-2+deb11u1 libjemalloc-dev=5.2.1-3
77

8-
# Add target architecture argument with default value
98
ARG RUST_TARGET="x86_64-unknown-linux-gnu"
109

1110
# Copy the project to the container
12-
COPY . /app
11+
COPY ./ /app
1312
WORKDIR /app
1413

15-
# Get the latest commit timestamp and set SOURCE_DATE_EPOCH (default it to 0 if not passed)
16-
ARG SOURCE_DATE=0
17-
18-
# Set environment variables for reproducibility
19-
ARG RUSTFLAGS="-C link-arg=-Wl,--build-id=none -C metadata='' --remap-path-prefix $(pwd)=."
20-
ENV SOURCE_DATE_EPOCH=$SOURCE_DATE \
21-
CARGO_INCREMENTAL=0 \
22-
LC_ALL=C \
23-
TZ=UTC \
24-
RUSTFLAGS="${RUSTFLAGS}"
25-
26-
# Set the default features if not provided
27-
ARG FEATURES="gnosis,slasher-lmdb,slasher-mdbx,slasher-redb,jemalloc"
28-
29-
# Set the default profile if not provided
30-
ARG PROFILE="reproducible"
31-
3214
# Build the project with the reproducible settings
33-
RUN cargo build --bin lighthouse \
34-
--features "${FEATURES}" \
35-
--profile "${PROFILE}" \
36-
--locked \
37-
--target "${RUST_TARGET}"
15+
RUN make build-reproducible
3816

39-
RUN mv /app/target/${RUST_TARGET}/${PROFILE}/lighthouse /lighthouse
17+
# Move the binary to a standard location
18+
RUN mv /app/target/${RUST_TARGET}/release/lighthouse /lighthouse
4019

4120
# Create a minimal final image with just the binary
4221
FROM gcr.io/distroless/cc-debian12:nonroot-6755e21ccd99ddead6edc8106ba03888cbeed41a
4322
COPY --from=builder /lighthouse /lighthouse
23+
4424
ENTRYPOINT [ "/lighthouse" ]

‎Makefile‎

Lines changed: 46 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -81,36 +81,67 @@ build-lcli-aarch64:
8181
build-lcli-riscv64:
8282
cross build --bin lcli --target riscv64gc-unknown-linux-gnu --features "portable" --profile "$(CROSS_PROFILE)" --locked
8383

84-
# extracts the current source date for reproducible builds
85-
SOURCE_DATE := $(shell git log -1 --pretty=%ct)
86-
87-
# Default image for x86_64
84+
# Environment variables for reproducible builds
85+
# Initialize RUSTFLAGS
86+
RUST_BUILD_FLAGS =
87+
# Remove build ID from the binary to ensure reproducibility across builds
88+
RUST_BUILD_FLAGS += -C link-arg=-Wl,--build-id=none
89+
# Remove metadata hash from symbol names to ensure reproducible builds
90+
RUST_BUILD_FLAGS += -C metadata=''
91+
92+
# Set timestamp from last git commit for reproducible builds
93+
SOURCE_DATE ?= $(shell git log -1 --pretty=%ct)
94+
95+
# Disable incremental compilation to avoid non-deterministic artifacts
96+
CARGO_INCREMENTAL_VAL = 0
97+
# Set C locale for consistent string handling and sorting
98+
LOCALE_VAL = C
99+
# Set UTC timezone for consistent time handling across builds
100+
TZ_VAL = UTC
101+
102+
# Features for reproducible builds
103+
FEATURES_REPRODUCIBLE = $(CROSS_FEATURES),jemalloc-unprefixed
104+
105+
# Derive the architecture-specific library path from RUST_TARGET
106+
JEMALLOC_LIB_ARCH = $(word 1,$(subst -, ,$(RUST_TARGET)))
107+
JEMALLOC_OVERRIDE = /usr/lib/$(JEMALLOC_LIB_ARCH)-linux-gnu/libjemalloc.a
108+
109+
# Default target architecture
110+
RUST_TARGET ?= x86_64-unknown-linux-gnu
111+
112+
# Default images for different architectures
88113
RUST_IMAGE_AMD64 ?= rust:1.88-bullseye@sha256:8e3c421122bf4cd3b2a866af41a4dd52d87ad9e315fd2cb5100e87a7187a9816
114+
RUST_IMAGE_ARM64 ?= rust:1.88-bullseye@sha256:8b22455a7ce2adb1355067638284ee99d21cc516fab63a96c4514beaf370aa94
89115

90-
# Reproducible build for x86_64
91-
build-reproducible-x86_64:
116+
.PHONY: build-reproducible
117+
build-reproducible: ## Build the lighthouse binary into `target` directory with reproducible builds
118+
SOURCE_DATE_EPOCH=$(SOURCE_DATE) \
119+
RUSTFLAGS="${RUST_BUILD_FLAGS} --remap-path-prefix $$(pwd)=." \
120+
CARGO_INCREMENTAL=${CARGO_INCREMENTAL_VAL} \
121+
LC_ALL=${LOCALE_VAL} \
122+
TZ=${TZ_VAL} \
123+
JEMALLOC_OVERRIDE=${JEMALLOC_OVERRIDE} \
124+
cargo build --bin lighthouse --features "$(FEATURES_REPRODUCIBLE)" --profile "$(PROFILE)" --locked --target $(RUST_TARGET)
125+
126+
.PHONY: build-reproducible-x86_64
127+
build-reproducible-x86_64: ## Build reproducible x86_64 Docker image
92128
DOCKER_BUILDKIT=1 docker build \
93129
--build-arg RUST_TARGET="x86_64-unknown-linux-gnu" \
94130
--build-arg RUST_IMAGE=$(RUST_IMAGE_AMD64) \
95-
--build-arg SOURCE_DATE=$(SOURCE_DATE) \
96131
-f Dockerfile.reproducible \
97132
-t lighthouse:reproducible-amd64 .
98133

99-
# Default image for arm64
100-
RUST_IMAGE_ARM64 ?= rust:1.88-bullseye@sha256:8b22455a7ce2adb1355067638284ee99d21cc516fab63a96c4514beaf370aa94
101-
102-
# Reproducible build for aarch64
103-
build-reproducible-aarch64:
134+
.PHONY: build-reproducible-aarch64
135+
build-reproducible-aarch64: ## Build reproducible aarch64 Docker image
104136
DOCKER_BUILDKIT=1 docker build \
105137
--platform linux/arm64 \
106138
--build-arg RUST_TARGET="aarch64-unknown-linux-gnu" \
107139
--build-arg RUST_IMAGE=$(RUST_IMAGE_ARM64) \
108-
--build-arg SOURCE_DATE=$(SOURCE_DATE) \
109140
-f Dockerfile.reproducible \
110141
-t lighthouse:reproducible-arm64 .
111142

112-
# Build both architectures
113-
build-reproducible-all: build-reproducible-x86_64 build-reproducible-aarch64
143+
.PHONY: build-reproducible-all
144+
build-reproducible-all: build-reproducible-x86_64 build-reproducible-aarch64 ## Build both x86_64 and aarch64 reproducible Docker images
114145

115146
# Create a `.tar.gz` containing a binary for a specific target.
116147
define tarball_release_binary

‎common/malloc_utils/Cargo.toml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,8 @@ jemalloc-profiling = ["tikv-jemallocator/profiling"]
2121
# Force the use of system malloc (or glibc) rather than jemalloc.
2222
# This is a no-op on Windows where jemalloc is always disabled.
2323
sysmalloc = []
24+
# Enable jemalloc with unprefixed malloc (recommended for reproducible builds)
25+
jemalloc-unprefixed = ["jemalloc", "tikv-jemallocator/unprefixed_malloc_on_supported_platforms"]
2426

2527
[dependencies]
2628
libc = "0.2.79"

‎testing/state_transition_vectors/Makefile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,4 +5,4 @@ test:
55
cargo test --release --features "$(TEST_FEATURES)"
66

77
clean:
8-
rm -r vectors/
8+
rm -rf vectors/

0 commit comments

Comments
 (0)