Skip to content

Commit 3a27a00

Browse files
willieyzwillieyz
committed
Port ec2_compatibilitytests from mlkem
- This commit port ec2_compatibilitytests from mlkem to mldsa - Also, due to the requement for ci_ec2_container.yml, this commit add the ci_ec2_container.yml based on ci_ec2_reuseable.yml, referencing from mlkem-native - About the AMI and AWS_ROLE we use, this commit aligns the AWS_ROLE and other parameters with those used in mlkem-native. we now use the same AMI, AWS_ROLE, and related configuration parameters for ec2_compatibilitytests to ensure consistency with mlkem-native. - Skip ACVP tests on Amazon Linux 2 containers when Python on these containers does not support hashlib sha512_224. - when adding several containers to ec2_compatibilitytests, we found that three Amazon Linux 2 containers could not run ACVP tests correctly due to missing support for newer hashlib hash types: * `amazonlinux-2-aarch:base` * `amazonlinux-2-aarch:gcc-7x` * `amazonlinux-2-aarch:clang-7x` - For these containers, we set `acvptest` to false. We also added a guard around `make quickcheck` in `ci_ec2_container.yml` and left a comment for future refinement. Signed-off-by: willieyz <willie.zhao@chelpis.com>
1 parent dbebfc6 commit 3a27a00

File tree

3 files changed

+310
-0
lines changed

3 files changed

+310
-0
lines changed

‎.github/actions/setup-yum/action.yml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# Copyright (c) The mlkem-native project authors
2+
# Copyright (c) The mldsa-native project authors
3+
# SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT
4+
5+
name: Dependencies (yum)
6+
description: Install dependencies via yum
7+
8+
inputs:
9+
packages:
10+
description: Space-separated list of additional packages to install
11+
required: false
12+
default: ''
13+
sudo:
14+
required: false
15+
default: 'sudo'
16+
17+
runs:
18+
using: composite
19+
steps:
20+
- name: Install base packages
21+
shell: bash
22+
run: |
23+
${{ inputs.sudo }} yum install make gcc python3 git -y
24+
- name: Install additional packages
25+
if: ${{ inputs.packages != ''}}
26+
shell: bash
27+
run: |
28+
${{ inputs.sudo }} yum install ${{ inputs.packages }} -y

‎.github/workflows/ci.yml‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -613,6 +613,74 @@ jobs:
613613
nix-shell: ""
614614
custom_shell: ${{ matrix.container.nix_shell && format('{0} --run \"bash -e {{0}}\"', matrix.container.nix_shell) || 'bash' }}
615615
gh_token: ${{ secrets.GITHUB_TOKEN }}
616+
ec2_compatibilitytests:
617+
strategy:
618+
max-parallel: 8
619+
fail-fast: false
620+
matrix:
621+
container:
622+
- id: amazonlinux-2-aarch:base
623+
# TODO: Python 3.7 on Amazon Linux 2 lacks `sha512_224` in hashlib; set to false to skip acvp.
624+
quickcheck: false
625+
- id: amazonlinux-2-aarch:gcc-7x
626+
# TODO: Python 3.7 on Amazon Linux 2 lacks `sha512_224` in hashlib; set to false to skip acvp.
627+
quickcheck: false
628+
- id: amazonlinux-2-aarch:clang-7x
629+
# TODO: Python 3.7 on Amazon Linux 2 lacks `sha512_224` in hashlib; set to false to skip acvp.
630+
quickcheck: false
631+
- id: amazonlinux-2023-aarch:base
632+
quickcheck: true
633+
- id: amazonlinux-2023-aarch:gcc-11x
634+
quickcheck: true
635+
- id: amazonlinux-2023-aarch:clang-15x
636+
quickcheck: true
637+
- id: amazonlinux-2023-aarch:clang-15x-sanitizer
638+
quickcheck: true
639+
# - id: amazonlinux-2023-aarch:cryptofuzz Not yet supported
640+
- id: ubuntu-22.04-aarch:gcc-12x
641+
quickcheck: true
642+
- id: ubuntu-22.04-aarch:gcc-11x
643+
quickcheck: true
644+
- id: ubuntu-20.04-aarch:gcc-8x
645+
quickcheck: true
646+
- id: ubuntu-20.04-aarch:gcc-7x
647+
quickcheck: true
648+
- id: ubuntu-20.04-aarch:clang-9x
649+
quickcheck: true
650+
- id: ubuntu-20.04-aarch:clang-8x
651+
quickcheck: true
652+
- id: ubuntu-20.04-aarch:clang-7x-bm-framework
653+
quickcheck: true
654+
- id: ubuntu-20.04-aarch:clang-7x
655+
quickcheck: true
656+
- id: ubuntu-20.04-aarch:clang-10x
657+
quickcheck: true
658+
- id: ubuntu-22.04-aarch:base
659+
quickcheck: true
660+
- id: ubuntu-20.04-aarch:base
661+
quickcheck: true
662+
name: Compatibility tests (${{ matrix.container.id }})
663+
permissions:
664+
contents: 'read'
665+
id-token: 'write'
666+
uses: ./.github/workflows/ci_ec2_container.yml
667+
if: github.repository_owner == 'pq-code-package' && !github.event.pull_request.head.repo.fork
668+
with:
669+
container: ${{ matrix.container.id }}
670+
name: ${{ matrix.container.id }}
671+
ec2_instance_type: t4g.small
672+
ec2_ami: ubuntu-latest (custom AMI)
673+
ec2_ami_id: ami-0c9bc1901ef0d1066 # Has docker images preinstalled
674+
compile_mode: native
675+
opt: all
676+
functest: true
677+
kattest: true
678+
acvptest: true
679+
quickcheck: ${{ matrix.container.quickcheck }}
680+
lint: false
681+
verbose: true
682+
cflags: "-O0"
683+
secrets: inherit
616684
check_autogenerated_files:
617685
strategy:
618686
fail-fast: false

‎.github/workflows/ci_ec2_container.yml‎

Lines changed: 214 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,214 @@
1+
# Copyright (c) The mlkem-native project authors
2+
# Copyright (c) The mldsa-native project authors
3+
# SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT
4+
5+
name: ci-ec2-reusable
6+
permissions:
7+
contents: read
8+
on:
9+
workflow_call:
10+
inputs:
11+
name:
12+
type: string
13+
description: Alternative name of instance
14+
default: Graviton2
15+
ec2_instance_type:
16+
type: string
17+
description: Type if EC2 instance to benchmark on
18+
default: t4g.small
19+
ec2_ami:
20+
type: string
21+
description: Textual description of AMI
22+
default: ubuntu-latest (aarch64)
23+
ec2_ami_id:
24+
type: string
25+
description: AMI ID
26+
default: ami-0e8c824f386e1de06
27+
cflags:
28+
type: string
29+
description: Custom CFLAGS for compilation
30+
default: ""
31+
verbose:
32+
description: Determine for the log verbosity
33+
type: boolean
34+
default: false
35+
compile_mode:
36+
type: string
37+
description: either all, native, cross or none
38+
default: all
39+
opt:
40+
type: string
41+
description: either all, opt or no_opt
42+
default: all
43+
functest:
44+
type: boolean
45+
default: true
46+
kattest:
47+
type: boolean
48+
default: true
49+
acvptest:
50+
type: boolean
51+
default: true
52+
quickcheck:
53+
type: boolean
54+
default: true
55+
lint:
56+
type: boolean
57+
default: true
58+
cbmc:
59+
type: boolean
60+
default: false
61+
cbmc_mldsa_parameter_set:
62+
type: string
63+
default: 44
64+
container:
65+
type: string
66+
default: ''
67+
env:
68+
AWS_ROLE: arn:aws:iam::559050233797:role/mlkem-c-aarch64-gh-action
69+
AWS_REGION: us-east-1
70+
AMI_UBUNTU_LATEST_X86_64: ami-0e86e20dae9224db8
71+
AMI_UBUNTU_LATEST_AARCH64: ami-096ea6a12ea24a797
72+
jobs:
73+
start-ec2-runner:
74+
name: Start instance (${{ inputs.ec2_instance_type }})
75+
permissions:
76+
contents: 'read'
77+
id-token: 'write'
78+
runs-on: ubuntu-latest
79+
if: ${{ always() }} # The point is to make this step non-cancellable,
80+
# avoiding race conditions where an instance is started,
81+
# but isn't yet done registering as a runner and reporting back.
82+
outputs:
83+
label: ${{ steps.remember-runner.outputs.label }}
84+
ec2-instance-id: ${{ steps.remember-runner.outputs.ec2-instance-id }}
85+
steps:
86+
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
87+
- name: Determine AMI ID
88+
id: det_ami_id
89+
run: |
90+
if [[ "${{ inputs.ec2_ami }}" == "ubuntu-latest (x86_64)" ]]; then
91+
AMI_ID=${{ env.AMI_UBUNTU_LATEST_X86_64 }}
92+
elif [[ "${{ inputs.ec2_ami }}" == "ubuntu-latest (aarch64)" ]]; then
93+
AMI_ID=${{ env.AMI_UBUNTU_LATEST_AARCH64 }}
94+
elif [[ "${{ inputs.ec2_ami }}" == "ubuntu-latest (custom AMI)" ]]; then
95+
AMI_ID=${{ inputs.ec2_ami_id }}
96+
fi
97+
echo "Using AMI ID: $AMI_ID"
98+
echo "AMI_ID=$AMI_ID" >> $GITHUB_OUTPUT
99+
- name: Configure AWS credentials
100+
uses: aws-actions/configure-aws-credentials@00943011d9042930efac3dcd3a170e4273319bc8 # v5.1.0
101+
with:
102+
role-to-assume: ${{ env.AWS_ROLE }}
103+
aws-region: ${{ env.AWS_REGION }}
104+
- name: Start EC2 runner
105+
id: start-ec2-runner-first
106+
continue-on-error: true
107+
uses: machulav/ec2-github-runner@a6dbcefcf8a31a861f5e078bb153ed332130c512 # v2.4.3
108+
with:
109+
mode: start
110+
github-token: ${{ secrets.AWS_GITHUB_TOKEN }}
111+
ec2-image-id: ${{ steps.det_ami_id.outputs.AMI_ID }}
112+
ec2-instance-type: ${{ inputs.ec2_instance_type }}
113+
subnet-id: subnet-07b2729e5e065962f
114+
security-group-id: sg-0ab2e297196c8c381
115+
- name: Start EC2 runner (wait before retry)
116+
if: steps.start-ec2-runner-first.outcome == 'failure'
117+
shell: bash
118+
run: |
119+
sleep 30 # Wait 30s before retrying
120+
sleep $((1 + RANDOM % 30))
121+
- name: Start EC2 runner (retry)
122+
id: start-ec2-runner-second
123+
if: steps.start-ec2-runner-first.outcome == 'failure'
124+
uses: machulav/ec2-github-runner@a6dbcefcf8a31a861f5e078bb153ed332130c512 # v2.4.3
125+
with:
126+
mode: start
127+
github-token: ${{ secrets.AWS_GITHUB_TOKEN }}
128+
ec2-image-id: ${{ steps.det_ami_id.outputs.AMI_ID }}
129+
ec2-instance-type: ${{ inputs.ec2_instance_type }}
130+
subnet-id: subnet-07b2729e5e065962f
131+
security-group-id: sg-0ab2e297196c8c381
132+
- name: Remember runner
133+
id: remember-runner
134+
shell: bash
135+
run: |
136+
if [[ "${{ steps.start-ec2-runner-first.outcome }}" == "failure" ]]; then
137+
echo "label=${{ steps.start-ec2-runner-second.outputs.label }}" >> "$GITHUB_OUTPUT"
138+
echo "ec2-instance-id=${{ steps.start-ec2-runner-second.outputs.ec2-instance-id }}" >> "$GITHUB_OUTPUT"
139+
else
140+
echo "label=${{ steps.start-ec2-runner-first.outputs.label }}" >> "$GITHUB_OUTPUT"
141+
echo "ec2-instance-id=${{ steps.start-ec2-runner-first.outputs.ec2-instance-id }}" >> "$GITHUB_OUTPUT"
142+
fi
143+
144+
tests:
145+
name: Run tests
146+
needs: start-ec2-runner
147+
if: ${{ inputs.container != '' }}
148+
runs-on: ${{ needs.start-ec2-runner.outputs.label }}
149+
container:
150+
localhost:5000/${{ inputs.container }}
151+
steps:
152+
# We're not using the checkout action here because on it's not supported
153+
# on all containers we want to test. Resort to a manual checkout.
154+
#
155+
# We can't hoist this into an action since calling an action can only
156+
# be done after checkout.
157+
- name: Manual checkout
158+
shell: bash
159+
run: |
160+
if /usr/bin/which yum; then
161+
yum install git -y
162+
elif /usr/bin/which apt; then
163+
apt update
164+
apt install git -y
165+
fi
166+
167+
git init
168+
git remote add origin $GITHUB_SERVER_URL/$GITHUB_REPOSITORY
169+
git fetch origin --depth 1 $GITHUB_SHA
170+
git checkout FETCH_HEAD
171+
- uses: ./.github/actions/setup-os
172+
with:
173+
sudo: ""
174+
- name: make quickcheck
175+
# Some containers lacks of support for hashlib support in acvp_client.py test; set to false to skip acvp in make quickcheck.
176+
if: ${{ inputs.quickcheck }}
177+
run: |
178+
OPT=0 make quickcheck
179+
make clean >/dev/null
180+
OPT=1 make quickcheck
181+
- name: Functional Tests
182+
uses: ./.github/actions/multi-functest
183+
with:
184+
nix-shell: ""
185+
gh_token: ${{ secrets.AWS_GITHUB_TOKEN }}
186+
cflags: ${{ inputs.cflags }}
187+
compile_mode: ${{ inputs.compile_mode }}
188+
opt: ${{ inputs.opt }}
189+
func: ${{ inputs.functest }}
190+
kat: ${{ inputs.kattest }}
191+
acvp: ${{ inputs.acvptest }}
192+
stop-ec2-runner:
193+
name: Stop instance (${{ inputs.ec2_instance_type }})
194+
permissions:
195+
contents: 'read'
196+
id-token: 'write'
197+
needs:
198+
- start-ec2-runner
199+
- tests
200+
runs-on: ubuntu-latest
201+
if: ${{ always() }} # required to stop the runner even if the error happened in the previous jobs
202+
steps:
203+
- name: Configure AWS credentials
204+
uses: aws-actions/configure-aws-credentials@00943011d9042930efac3dcd3a170e4273319bc8 # v5.1.0
205+
with:
206+
role-to-assume: ${{ env.AWS_ROLE }}
207+
aws-region: ${{ env.AWS_REGION }}
208+
- name: Stop EC2 runner
209+
uses: machulav/ec2-github-runner@a6dbcefcf8a31a861f5e078bb153ed332130c512 # v2.4.3
210+
with:
211+
mode: stop
212+
github-token: ${{ secrets.AWS_GITHUB_TOKEN }}
213+
label: ${{ needs.start-ec2-runner.outputs.label }}
214+
ec2-instance-id: ${{ needs.start-ec2-runner.outputs.ec2-instance-id }}

0 commit comments

Comments
 (0)