Fix kernel module signing with ephemeral keys for official builds #3493

danzatt · 2025-11-14T11:35:14Z

Summary

This PR fixes kernel module signing to work correctly with ephemeral keys for official builds while maintaining persistent keys for development builds. It also resolves portage sandbox violations that occurred when building kernel modules.

Problem

After switching to ephemeral keys (in /tmp) for official builds, two issues emerged:

Overly strict validation: The get_sig_key() function was enforcing /tmp/* paths for ALL builds, but this check should only apply to official builds.
Sandbox violations: For development builds using persistent keys in /home/sdk/.module-signing-keys, portage's sandbox was blocking write access during the setup_keys() function, causing build failures.

Solution

Commit 1: Fix ephemeral key directory paths baked into container images

Made the /tmp enforcement conditional on COREOS_OFFICIAL=1
Development builds can now use persistent directories outside /tmp
Added runtime detection of stale temp directories in sdk_entry.sh
Cleaned up ephemeral key variables from SDK container images during build

Commit 2: Add portage sandbox exception for development builds

Added addwrite "${MODULE_SIGNING_KEY_DIR}" in setup_keys() for unofficial builds
This allows portage to write to /home/sdk/.module-signing-keys during development
Official builds using /tmp don't need this exception (already writable)

Testing

✅ Official builds now use ephemeral keys in /tmp successfully
✅ Development builds use persistent keys in /home/sdk/.module-signing-keys without sandbox violations
✅ Module signing works for both in-tree and out-of-tree kernel modules

Files Changed

sdk_container/src/third_party/coreos-overlay/eclass/coreos-kernel.eclass: Added conditional /tmp check and sandbox exception
sdk_lib/sdk_entry.sh: Added stale directory detection and conditional key directory creation
sdk_lib/Dockerfile.sdk-build: Clean up ephemeral variables during build
sdk_lib/Dockerfile.sdk-import: Clean up ephemeral variables during build
sdk_lib/Dockerfile.sdk-update: Clean up ephemeral variables during build

Signed-off-by: Daniel Zatovic daniel.zatovic@gmail.com

krnowak · 2025-11-14T13:47:16Z

sdk_lib/sdk_entry.sh

+# Check if MODULE_SIGNING_KEY_DIR exists in .bashrc and if the directory actually exists
+if grep -q 'export MODULE_SIGNING_KEY_DIR' /home/sdk/.bashrc; then
+    # Extract the existing path
+    EXISTING_DIR=$(grep 'export MODULE_SIGNING_KEY_DIR' /home/sdk/.bashrc | sed "s/.*MODULE_SIGNING_KEY_DIR='\(.*\)'/\1/")


You probably could use cut like:

Suggested change

EXISTING_DIR=$(grep 'export MODULE_SIGNING_KEY_DIR' /home/sdk/.bashrc | sed "s/.*MODULE_SIGNING_KEY_DIR='$.*$'/\1/")

EXISTING_DIR=$(grep 'export MODULE_SIGNING_KEY_DIR=' /home/sdk/.bashrc | cut -f2- -d=)

krnowak · 2025-11-14T13:48:16Z

sdk_lib/sdk_entry.sh

+    # Extract the existing path
+    EXISTING_DIR=$(grep 'export MODULE_SIGNING_KEY_DIR' /home/sdk/.bashrc | sed "s/.*MODULE_SIGNING_KEY_DIR='\(.*\)'/\1/")
+    # If directory doesn't exist (stale from image build), remove the old entries and recreate
+    if [[ ! -d "$EXISTING_DIR" ]]; then


No need to quote variables inside double brackets in bash:

Suggested change

if [[ ! -d "$EXISTING_DIR" ]]; then

if [[ ! -d ${EXISTING_DIR} ]]; then

krnowak · 2025-11-14T13:51:36Z

sdk_lib/sdk_entry.sh

+        sed -i '/export MODULE_SIGNING_KEY_DIR/d' /home/sdk/.bashrc
+        sed -i '/export MODULES_SIGN_KEY/d' /home/sdk/.bashrc
+        sed -i '/export MODULES_SIGN_CERT/d' /home/sdk/.bashrc


This could be a single invocation to sed, like:

(Also I'd add = to the keys to be sure we match the exact key, not like MODULE_SIGNING_KEY_DIR_HAHA_MY_OWN).

Suggested change

sed -i '/export MODULE_SIGNING_KEY_DIR/d' /home/sdk/.bashrc

sed -i '/export MODULES_SIGN_KEY/d' /home/sdk/.bashrc

sed -i '/export MODULES_SIGN_CERT/d' /home/sdk/.bashrc

sed -i -e '/export MODULE_SIGNING_KEY_DIR=/d' \

-e '/export MODULES_SIGN_KEY=/d' \

-e '/export MODULES_SIGN_CERT=/d' /home/sdk/.bashrc

krnowak · 2025-11-14T13:53:55Z

sdk_lib/Dockerfile.sdk-build

+RUN sed -i '/export MODULE_SIGNING_KEY_DIR/d' /home/sdk/.bashrc && \
+    sed -i '/export MODULES_SIGN_KEY/d' /home/sdk/.bashrc && \
+    sed -i '/export MODULES_SIGN_CERT/d' /home/sdk/.bashrc


This could be a single sed invocation, and I'd add = to keys, see other comment below for details.

krnowak · 2025-11-14T13:54:00Z

sdk_lib/Dockerfile.sdk-import

+RUN sed -i '/export MODULE_SIGNING_KEY_DIR/d' /home/sdk/.bashrc && \
+    sed -i '/export MODULES_SIGN_KEY/d' /home/sdk/.bashrc && \
+    sed -i '/export MODULES_SIGN_CERT/d' /home/sdk/.bashrc


This could be a single sed invocation, and I'd add = to keys, see other comment below for details.

krnowak · 2025-11-14T13:54:07Z

sdk_lib/Dockerfile.sdk-update

+RUN sed -i '/export MODULE_SIGNING_KEY_DIR/d' /home/sdk/.bashrc && \
+    sed -i '/export MODULES_SIGN_KEY/d' /home/sdk/.bashrc && \
+    sed -i '/export MODULES_SIGN_CERT/d' /home/sdk/.bashrc


This could be a single sed invocation, and I'd add = to keys, see other comment below for details.

krnowak · 2025-11-14T13:54:26Z

sdk_lib/sdk_entry.sh

-grep -q 'export MODULE_SIGNING_KEY_DIR' /home/sdk/.bashrc || {
-    MODULE_SIGNING_KEY_DIR=$(su sdk -c "mktemp -d")
+# Check if MODULE_SIGNING_KEY_DIR exists in .bashrc and if the directory actually exists
+if grep -q 'export MODULE_SIGNING_KEY_DIR' /home/sdk/.bashrc; then


Suggested change

if grep -q 'export MODULE_SIGNING_KEY_DIR' /home/sdk/.bashrc; then

if grep -q 'export MODULE_SIGNING_KEY_DIR=' /home/sdk/.bashrc; then

krnowak · 2025-11-14T13:54:45Z

sdk_lib/sdk_entry.sh

+fi
+
+# Create key directory if not already configured in .bashrc
+if ! grep -q 'export MODULE_SIGNING_KEY_DIR' /home/sdk/.bashrc; then


Suggested change

if ! grep -q 'export MODULE_SIGNING_KEY_DIR' /home/sdk/.bashrc; then

if ! grep -q 'export MODULE_SIGNING_KEY_DIR=' /home/sdk/.bashrc; then

krnowak · 2025-11-14T13:55:20Z

sdk_lib/sdk_entry.sh

+        MODULE_SIGNING_KEY_DIR=$(su sdk -c "mktemp -d")
+    else
+        MODULE_SIGNING_KEY_DIR="/home/sdk/.module-signing-keys"
+        su sdk -c "mkdir -p '$MODULE_SIGNING_KEY_DIR'"


Use quoting feature of bash (search for ${parameter@operator} in https://www.gnu.org/software/bash/manual/bash.html#Shell-Parameter-Expansion):

Suggested change

su sdk -c "mkdir -p '$MODULE_SIGNING_KEY_DIR'"

su sdk -c "mkdir -p ${MODULE_SIGNING_KEY_DIR@Q}"

danzatt · 2025-11-17T14:44:39Z

Thanks. Applied all review suggestions from @krnowak (consolidated sed calls, added = to patterns, fixed indentation, quoted variables, improved variable extraction method).

Also I've discovered a critical bug: COREOS_OFFICIAL wasn't available during sdk_entry.sh execution, causing official builds to incorrectly use persistent keys instead of ephemeral /tmp keys. Fixed by sourcing /mnt/host/source/.sdkenv at the beginning of sdk_entry.sh.

Test:

# Official build - should use /tmp
COREOS_OFFICIAL=1 ./run_sdk_container -t --rm
# In container: grep MODULE_SIGNING_KEY_DIR ~/.bashrc
# echo $MODULE_SIGNING_KEY_DIR
# Expected: /tmp/tmp.XXXXXXXX

# Dev build - should use persistent dir
./run_sdk_container -t --rm
# Expected: /home/sdk/.module-signing-keys

sdk_container/src/third_party/coreos-overlay/eclass/coreos-kernel.eclass

danzatt · 2025-11-17T15:12:12Z

Also, I am not quite sure how to properly test the COREOS_OFFICIAL. In official builds, it seems to be passed also in the sdk_container/.repo/manifests/version.txt Should I also include that?

danzatt · 2025-11-19T09:16:53Z

CI is failing due to missing bincache artifacts (due to bincache migration). However I've rebased my #3162 onto this branch and the tests passed there yesterday, so I think we're good. If there are no objections, I'll merge.

chewi · 2025-11-19T10:33:53Z

Also, I am not quite sure how to properly test the COREOS_OFFICIAL. In official builds, it seems to be passed also in the sdk_container/.repo/manifests/version.txt Should I also include that?

I don't think it's passed in that file but determined by the format of FLATCAR_VERSION via the is_official function. Whenever I've needed to test a "pretend" official build, I've just temporarily adjusted the places where COREOS_OFFICIAL is checked.

tormath1 · 2025-11-19T11:05:25Z

CI is failing due to missing bincache artifacts (due to bincache migration). However I've rebased my #3162 onto this branch and the tests passed there yesterday, so I think we're good. If there are no objections, I'll merge.

FWIW, one can still access the old bincache artifacts while the instance is still up: http://145.40.90.147/

The SDK container build process was persisting temporary directory paths for module signing keys into /home/sdk/.bashrc. This caused all container instances to share the same ephemeral key location. Fixed by: - Runtime check in sdk_entry.sh to recreate stale temp directories - Build-time cleanup in Dockerfiles to remove the variables Each container instance now gets unique temporary directories. Signed-off-by: Daniel Zatovic <daniel.zatovic@gmail.com>

For official builds (COREOS_OFFICIAL=1), continue using ephemeral temporary directories for module signing keys. For unofficial/development builds, use a persistent directory at /mnt/host/source/.module-signing-keys to preserve keys across container restarts. Signed-off-by: Daniel Zatovic <daniel.zatovic@gmail.com>

…erns in Dockerfile.sdk-update Signed-off-by: Daniel Zatovic <daniel.zatovic@gmail.com>

Signed-off-by: Daniel Zatovic <daniel.zatovic@gmail.com>

Fix bug where COREOS_OFFICIAL wasn't available during sdk_entry.sh execution, causing official builds to incorrectly use persistent module signing keys instead of ephemeral /tmp keys. Signed-off-by: Daniel Zatovic <daniel.zatovic@gmail.com>

danzatt · 2025-11-20T08:55:34Z

Jenkins job http://jenkins.infra.kinvolk.io:8080/job/container/job/packages_all_arches/7003 just succeeded. I'll merge, thanks for the review.

danzatt requested a review from a team as a code owner November 14, 2025 11:35

danzatt had a problem deploying to development November 14, 2025 11:35 — with GitHub Actions Error

krnowak reviewed Nov 14, 2025

View reviewed changes

danzatt temporarily deployed to development November 17, 2025 14:44 — with GitHub Actions Inactive

chewi reviewed Nov 17, 2025

View reviewed changes

sdk_container/src/third_party/coreos-overlay/eclass/coreos-kernel.eclass Show resolved Hide resolved

chewi approved these changes Nov 17, 2025

View reviewed changes

danzatt temporarily deployed to development November 19, 2025 12:15 — with GitHub Actions Inactive

danzatt added 5 commits November 19, 2025 13:17

Apply PR review suggestion: consolidate sed calls and add '=' to patt…

1dbd08e

…erns in Dockerfile.sdk-update Signed-off-by: Daniel Zatovic <daniel.zatovic@gmail.com>

eclass/coreos-kernel: fix indentation and codestyle

7d30c71

Signed-off-by: Daniel Zatovic <daniel.zatovic@gmail.com>

danzatt force-pushed the danzatt/ephemeral-keys-only-official branch from 71b3de3 to 5de9580 Compare November 19, 2025 13:26

danzatt temporarily deployed to development November 19, 2025 13:26 — with GitHub Actions Inactive

danzatt merged commit f05097d into flatcar:main Nov 20, 2025
7 of 9 checks passed

github-actions bot mentioned this pull request Nov 22, 2025

Monthly contributions report 2025-10-22 - 2025-11-21 flatcar/Flatcar#1953

Open

	EXISTING_DIR=$(grep 'export MODULE_SIGNING_KEY_DIR' /home/sdk/.bashrc \| sed "s/.MODULE_SIGNING_KEY_DIR='\(.\)'/\1/")
	EXISTING_DIR=$(grep 'export MODULE_SIGNING_KEY_DIR=' /home/sdk/.bashrc \| cut -f2- -d=)

	if [[ ! -d "$EXISTING_DIR" ]]; then
	if [[ ! -d ${EXISTING_DIR} ]]; then

	if grep -q 'export MODULE_SIGNING_KEY_DIR' /home/sdk/.bashrc; then
	if grep -q 'export MODULE_SIGNING_KEY_DIR=' /home/sdk/.bashrc; then

	if ! grep -q 'export MODULE_SIGNING_KEY_DIR' /home/sdk/.bashrc; then
	if ! grep -q 'export MODULE_SIGNING_KEY_DIR=' /home/sdk/.bashrc; then

	su sdk -c "mkdir -p '$MODULE_SIGNING_KEY_DIR'"
	su sdk -c "mkdir -p ${MODULE_SIGNING_KEY_DIR@Q}"

Fix kernel module signing with ephemeral keys for official builds #3493

Fix kernel module signing with ephemeral keys for official builds #3493

Uh oh!

Conversation

danzatt commented Nov 14, 2025

Summary

Problem

Solution

Commit 1: Fix ephemeral key directory paths baked into container images

Commit 2: Add portage sandbox exception for development builds

Testing

Files Changed

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danzatt commented Nov 17, 2025

Uh oh!

Uh oh!

danzatt commented Nov 17, 2025

Uh oh!

danzatt commented Nov 19, 2025

Uh oh!

chewi commented Nov 19, 2025

Uh oh!

tormath1 commented Nov 19, 2025

Uh oh!

danzatt commented Nov 20, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants