Skip to content

Commit 1649656

Browse files
astuyveastuyve
authored
feat: Automatic layer releases (#421)
* feat: Copy gitlab-ci from datadog-lambda-js * feat: first crack at migration the build script for individual publishing * feat: secrets for python. TODO - actually create them * feat: Add python runtimes * feat: re-organize architecture flags. Support separate arch/name * feat: first draft of build generator template * feat: Add datasources and first cut at publish_pypi script * feat: oops, no node. TODO: fix container names * feat: pass the right layer name to sign layers * feat: Python before script * feat: arch parameterized sign layer * feat: fix up runtimes * fix: build layer arch * fix: Check layer size arch * feat: zip file is py not python * fix: fix up check layer size script * hotfix: use js ssm secrets until I can figure out which AWS account CI runs in and can add them * feat: Less uniformity on images in python, specify in runtimes.yaml * feat: Can't use permissions across repos * feat: lol our ci runners set DD_SERVICE so it breaks our unit tests * feat: Gotta add yarn so we can add serverless * feat: lint * feat: Break install-node into separate task * feat: Use name instead of python_version * fix: no python-, just the version number * empty commit to bump CI * feat: integration tests should run for both architectures * fix: arg, no arch in integration test * fix: pass sls framework the proper arch * feat: fix script * feat: Default to x86 just so the invoke function works * fix: nvm pass the sls arch everywhere I guess * fix: strip arch from user agent * feat: the right x86_64 arch for serverless framework * fix: globalize local env * feat: fix regex * fix: ints should pass now * feat: Update tests * fix: lint * fix: lint * feat: lint * feat: I think we just need one lint * feat: remove install node for publish step * feat: remove integration tests from github build, we run them in gitlab now * feat: token applied
1 parent 17c8e0b commit 1649656

20 files changed

+691
-195
lines changed

‎.github/workflows/build.yml‎

Lines changed: 0 additions & 71 deletions
Original file line numberDiff line numberDiff line change
@@ -63,74 +63,3 @@ jobs:
6363
run: |
6464
source venv/bin/activate
6565
pytest -vv
66-
67-
integration-test:
68-
runs-on: ubuntu-latest
69-
strategy:
70-
matrix:
71-
runtime-param: ['3.8', '3.9', '3.10', '3.11', '3.12']
72-
steps:
73-
- name: Checkout
74-
uses: actions/checkout@v3
75-
76-
- name: Set up Node 14
77-
uses: actions/setup-node@v3
78-
with:
79-
node-version: 14
80-
81-
- name: Cache Node modules
82-
id: cache-node-modules
83-
uses: actions/cache@v3
84-
with:
85-
path: "**/node_modules"
86-
key: ${{ runner.os }}-modules-${{ hashFiles('**/yarn.lock') }}
87-
88-
- name: Set up Python
89-
uses: actions/setup-python@v4
90-
with:
91-
python-version: 3.9
92-
93-
- name: Install Python dependencies
94-
run: |
95-
pip install virtualenv
96-
virtualenv venv
97-
source venv/bin/activate
98-
pip install .[dev]
99-
100-
- name: Install Serverless Framework
101-
run: sudo yarn global add serverless@^3.7.0 --prefix /usr/local
102-
- name: Install Crossbuild Deps
103-
run: |
104-
sudo apt-get update --allow-releaseinfo-change --fix-missing
105-
sudo apt install -y qemu-user-static binfmt-support
106-
107-
- name: Install dependencies
108-
if: steps.cache-node-modules.outputs.cache-hit != 'true'
109-
working-directory: tests/integration
110-
run: yarn install
111-
112-
- name: Run tests
113-
env:
114-
BUILD_LAYERS: true
115-
DD_API_KEY: ${{ secrets.DD_API_KEY }}
116-
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
117-
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
118-
RUNTIME_PARAM: ${{ matrix.runtime-param }}
119-
run: ./scripts/run_integration_tests.sh
120-
121-
- name: Send success metric
122-
env:
123-
DD_API_KEY: ${{ secrets.DD_API_KEY }}
124-
run: ./scripts/send_status_metric.sh 0 $DD_API_KEY
125-
126-
integration-test-failure:
127-
runs-on: ubuntu-latest
128-
needs: [integration-test]
129-
if: always() && (needs.integration-test.result == 'failure')
130-
steps:
131-
- name: Checkout
132-
uses: actions/checkout@v3
133-
- name: Send a failure metric
134-
env:
135-
DD_API_KEY: ${{ secrets.DD_API_KEY }}
136-
run: ./scripts/send_status_metric.sh 1 $DD_API_KEY

‎.gitlab-ci.yml‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
stages:
2+
- pre
3+
- build
4+
5+
.go-cache: &go-cache
6+
key: datadog-lambda-python-go-cache
7+
policy: pull
8+
9+
generator:
10+
stage: pre
11+
image: registry.ddbuild.io/images/mirror/golang:alpine
12+
tags: ["arch:amd64"]
13+
cache: *go-cache
14+
script:
15+
- apk add --no-cache gomplate
16+
- gomplate --config ci/config.yaml
17+
artifacts:
18+
paths:
19+
- ci/*-pipeline.yaml
20+
21+
build-layers:
22+
stage: build
23+
trigger:
24+
include:
25+
- artifact: ci/build-pipeline.yaml
26+
job: generator
27+
strategy: depend
28+
rules:
29+
- when: on_success

‎ci/config.yaml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
inputFiles:
2+
- ci/input_files/build.yaml.tpl
3+
4+
outputFiles:
5+
- ci/build-pipeline.yaml
6+
7+
datasources:
8+
runtimes:
9+
url: ci/datasources/runtimes.yaml
10+
regions:
11+
url: ci/datasources/regions.yaml
12+
environments:
13+
url: ci/datasources/environments.yaml

‎ci/datasources/environments.yaml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
environments:
2+
- name: sandbox
3+
external_id: sandbox-publish-externalid
4+
role_to_assume: sandbox-layer-deployer
5+
account: 425362996713
6+
- name: prod
7+
external_id: prod-publish-externalid
8+
role_to_assume: dd-serverless-layer-deployer-role
9+
account: 464622532012

‎ci/datasources/regions.yaml‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
regions:
2+
- code: "us-east-1"
3+
- code: "us-east-2"
4+
- code: "us-west-1"
5+
- code: "us-west-2"
6+
- code: "af-south-1"
7+
- code: "ap-east-1"
8+
- code: "ap-south-1"
9+
- code: "ap-south-2"
10+
- code: "ap-southeast-1"
11+
- code: "ap-southeast-2"
12+
- code: "ap-southeast-3"
13+
- code: "ap-southeast-4"
14+
- code: "ap-northeast-1"
15+
- code: "ap-northeast-2"
16+
- code: "ap-northeast-3"
17+
- code: "ca-central-1"
18+
# - code: "ca-west-1" we don't support it
19+
- code: "eu-central-1"
20+
- code: "eu-central-2"
21+
- code: "eu-west-1"
22+
- code: "eu-west-2"
23+
- code: "eu-west-3"
24+
- code: "eu-south-1"
25+
- code: "eu-south-2"
26+
# - code: "il-central-1" we don't support it
27+
- code: "me-south-1"
28+
- code: "me-central-1"
29+
- code: "sa-east-1"

‎ci/datasources/runtimes.yaml‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
runtimes:
2+
- name: "python38"
3+
python_version: "3.8"
4+
arch: "amd64"
5+
image: "3.8"
6+
- name: "python38"
7+
python_version: "3.8"
8+
arch: "arm64"
9+
image: "3.8"
10+
- name: "python39"
11+
python_version: "3.8"
12+
arch: "amd64"
13+
image: "3.9"
14+
- name: "python39"
15+
python_version: "3.9"
16+
arch: "arm64"
17+
image: "3.9"
18+
- name: "python310"
19+
python_version: "3.10"
20+
arch: "amd64"
21+
image: "3.10"
22+
- name: "python310"
23+
python_version: "3.10"
24+
arch: "arm64"
25+
image: "3.10"
26+
- name: "python311"
27+
python_version: "3.11"
28+
arch: "amd64"
29+
image: "3.11.6"
30+
- name: "python311"
31+
python_version: "3.11"
32+
arch: "arm64"
33+
image: "3.11.6"
34+
- name: "python312"
35+
python_version: "3.12"
36+
arch: "amd64"
37+
image: "3.12.0"
38+
- name: "python312"
39+
python_version: "3.12"
40+
arch: "arm64"
41+
image: "3.12.0"

‎ci/get_secrets.sh‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
#!/bin/bash
2+
3+
# Unless explicitly stated otherwise all files in this repository are licensed
4+
# under the Apache License Version 2.0.
5+
# This product includes software developed at Datadog (https://www.datadoghq.com/).
6+
# Copyright 2023 Datadog, Inc.
7+
8+
set -e
9+
10+
if [ -z "$EXTERNAL_ID_NAME" ]; then
11+
printf "[Error] No EXTERNAL_ID_NAME found.\n"
12+
printf "Exiting script...\n"
13+
exit 1
14+
fi
15+
16+
if [ -z "$ROLE_TO_ASSUME" ]; then
17+
printf "[Error] No ROLE_TO_ASSUME found.\n"
18+
printf "Exiting script...\n"
19+
exit 1
20+
fi
21+
22+
printf "Getting AWS External ID...\n"
23+
24+
EXTERNAL_ID=$(aws ssm get-parameter \
25+
--region us-east-1 \
26+
--name "ci.datadog-lambda-python.$EXTERNAL_ID_NAME" \
27+
--with-decryption \
28+
--query "Parameter.Value" \
29+
--out text)
30+
31+
printf "Getting DD API KEY...\n"
32+
33+
export DD_API_KEY=$(aws ssm get-parameter \
34+
--region us-east-1 \
35+
--name ci.datadog-lambda-python.dd-api-key \
36+
--with-decryption \
37+
--query "Parameter.Value" \
38+
--out text)
39+
40+
printf "Assuming role...\n"
41+
42+
export $(printf "AWS_ACCESS_KEY_ID=%s AWS_SECRET_ACCESS_KEY=%s AWS_SESSION_TOKEN=%s" \
43+
$(aws sts assume-role \
44+
--role-arn "arn:aws:iam::$AWS_ACCOUNT:role/$ROLE_TO_ASSUME" \
45+
--role-session-name "ci.datadog-lambda-python-$CI_JOB_ID-$CI_JOB_STAGE" \
46+
--query "Credentials.[AccessKeyId,SecretAccessKey,SessionToken]" \
47+
--external-id $EXTERNAL_ID \
48+
--output text))

0 commit comments

Comments
 (0)