Skip to content

Commit 18f79ff

Browse files
MaciejKarasMaciejKaras
authored
CLOUDP-295785 - simplify pipeline.sh script and use it for all images (#543)
# Summary Combines all `pipeline.py` invocations into single `pipeline.sh` script which makes it easier to maintain and reduces amount of future bugs. The same principle was applied in `.evergreen-functions.yml`. Now we have only single `pipeline` evergreen function. For evergreen extensions I've decided to stick with lower-case variable naming. This is aligning better with existing evg functions + less conflicts with context variables. This also fixes the bug, where the `BUILD_SCENARIO: release` for OM and agent release tasks was not properly overriding the default one from `evg_private_context`. ## Proof of Work 1. Custom [rebuild_currently_used_agents](https://spruce.mongodb.com/task/mongodb_kubernetes_manual_release_agent_currently_used_rebuild_currently_used_agents_patch_fd4450bdd5aab857c0b9570cfd1cb4f3b8bb42f8_68f7626ffca1f10007b316df_25_10_21_10_37_36/logs?execution=0) task with flags was successful 2. [Image release tasks](https://spruce.mongodb.com/version/68f76366a6e778000773cdfd/tasks?page=0&sorts=STATUS%3AASC%3BBASE_STATUS%3ADESC&variant=%5Erelease_images%24) where successful and released images to quay.io with correct versions 3. Published OM and agent images using fixed [tasks](http://spruce.mongodb.com/version/68f76720b122440007395905/tasks?page=0&sorts=STATUS%3AASC%3BBASE_STATUS%3ADESC&variant=%5Epublish_om80_images%24) ## Checklist - [x] Have you linked a jira ticket and/or is the ticket in the title? - [x] Have you checked whether your jira ticket required DOCSP changes? - [x] Have you added changelog file? - use `skip-changelog` label if not needed - refer to [Changelog files and Release Notes](https://github.com/mongodb/mongodb-kubernetes/blob/master/CONTRIBUTING.md#changelog-files-and-release-notes) section in CONTRIBUTING.md for more details
1 parent f214fc8 commit 18f79ff

File tree

8 files changed

+89
-124
lines changed

8 files changed

+89
-124
lines changed

‎.evergreen-functions.yml‎

Lines changed: 4 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -507,7 +507,7 @@ functions:
507507
params:
508508
working_dir: src/github.com/mongodb/mongodb-kubernetes
509509
binary: scripts/release/publish_helm_chart.sh
510-
510+
511511
build_multi_cluster_binary:
512512
- command: subprocess.exec
513513
params:
@@ -542,50 +542,9 @@ functions:
542542
working_dir: src/github.com/mongodb/mongodb-kubernetes
543543
binary: scripts/release/pipeline.sh
544544
env:
545-
IMAGE_NAME: ${IMAGE_NAME}
546-
FLAGS: ${FLAGS}
547-
548-
pipeline_agent:
549-
- command: subprocess.exec
550-
retry_on_failure: true
551-
type: setup
552-
params:
553-
shell: bash
554-
<<: *e2e_include_expansions_in_env
555-
working_dir: src/github.com/mongodb/mongodb-kubernetes
556-
binary: scripts/release/pipeline_agent.sh
557-
env:
558-
FLAGS: ${FLAGS}
559-
560-
pipeline_ops_manager:
561-
- command: subprocess.exec
562-
retry_on_failure: true
563-
type: setup
564-
params:
565-
shell: bash
566-
<<: *e2e_include_expansions_in_env
567-
working_dir: src/github.com/mongodb/mongodb-kubernetes
568-
binary: scripts/release/pipeline_ops_manager.sh
569-
570-
pipeline_readiness_probe:
571-
- command: subprocess.exec
572-
retry_on_failure: true
573-
type: setup
574-
params:
575-
shell: bash
576-
<<: *e2e_include_expansions_in_env
577-
working_dir: src/github.com/mongodb/mongodb-kubernetes
578-
binary: scripts/release/pipeline_readiness_probe.sh
579-
580-
pipeline_version_upgrade_hook:
581-
- command: subprocess.exec
582-
retry_on_failure: true
583-
type: setup
584-
params:
585-
shell: bash
586-
<<: *e2e_include_expansions_in_env
587-
working_dir: src/github.com/mongodb/mongodb-kubernetes
588-
binary: scripts/release/pipeline_version_upgrade_hook.sh
545+
IMAGE_NAME: ${image_name}
546+
BUILD_SCENARIO_OVERRIDE: ${build_scenario}
547+
FLAGS: ${flags}
589548

590549
teardown_cloud_qa_all:
591550
- command: shell.exec

‎.evergreen-release.yml‎

Lines changed: 13 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ tasks:
1212
- func: build_multi_cluster_binary
1313
- func: pipeline
1414
vars:
15-
IMAGE_NAME: meko-tests
15+
image_name: meko-tests
1616

1717
- name: release_operator
1818
tags: [ "image_release" ]
@@ -23,7 +23,7 @@ tasks:
2323
- func: quay_login
2424
- func: pipeline
2525
vars:
26-
IMAGE_NAME: operator
26+
image_name: operator
2727

2828
# Releases init images to Quay
2929
- name: release_init_appdb
@@ -35,7 +35,7 @@ tasks:
3535
- func: quay_login
3636
- func: pipeline
3737
vars:
38-
IMAGE_NAME: init-appdb
38+
image_name: init-appdb
3939

4040
- name: release_init_database
4141
tags: [ "image_release" ]
@@ -46,7 +46,7 @@ tasks:
4646
- func: quay_login
4747
- func: pipeline
4848
vars:
49-
IMAGE_NAME: init-database
49+
image_name: init-database
5050

5151
- name: release_init_ops_manager
5252
tags: [ "image_release" ]
@@ -57,7 +57,7 @@ tasks:
5757
- func: quay_login
5858
- func: pipeline
5959
vars:
60-
IMAGE_NAME: init-ops-manager
60+
image_name: init-ops-manager
6161

6262
- name: release_database
6363
tags: [ "image_release" ]
@@ -68,7 +68,7 @@ tasks:
6868
- func: quay_login
6969
- func: pipeline
7070
vars:
71-
IMAGE_NAME: database
71+
image_name: database
7272

7373
- name: release_readiness_probe
7474
tags: [ "image_release" ]
@@ -77,7 +77,9 @@ tasks:
7777
- func: clone
7878
- func: setup_building_host
7979
- func: quay_login
80-
- func: pipeline_readiness_probe
80+
- func: pipeline
81+
vars:
82+
image_name: readiness-probe
8183

8284
- name: release_version_upgrade_hook
8385
tags: [ "image_release" ]
@@ -86,7 +88,9 @@ tasks:
8688
- func: clone
8789
- func: setup_building_host
8890
- func: quay_login
89-
- func: pipeline_version_upgrade_hook
91+
- func: pipeline
92+
vars:
93+
image_name: upgrade-hook
9094

9195
- name: prepare_and_upload_openshift_bundles
9296
tags: [ "openshift_bundles" ]
@@ -123,7 +127,7 @@ tasks:
123127
- func: install_macos_notarization_service
124128
- func: python_venv
125129
- func: release_kubectl_mongodb_plugin
126-
130+
127131
- name: create_chart_release_pr
128132
tags: [ "helm_chart_release_pr" ]
129133
commands:

‎.evergreen.yml‎

Lines changed: 37 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -285,9 +285,10 @@ tasks:
285285
- func: setup_building_host
286286
- func: quay_login
287287
- func: setup_docker_sbom
288-
- func: pipeline_agent
288+
- func: pipeline
289289
vars:
290-
BUILD_SCENARIO: release
290+
image_name: agent
291+
build_scenario: release
291292

292293
- name: migrate_all_agents
293294
# this enables us to run this variant manually to build all the agents for the new agent registry
@@ -296,10 +297,11 @@ tasks:
296297
- func: clone
297298
- func: setup_building_host
298299
- func: quay_login
299-
- func: pipeline_agent
300+
- func: pipeline
300301
vars:
301-
BUILD_SCENARIO: release
302-
FLAGS: "--all-agents -r quay.io/mongodb/mongodb-agent --skip-if-exists=false"
302+
image_name: agent
303+
build_scenario: release
304+
flags: "--parallel --all-agents -r quay.io/mongodb/mongodb-agent --skip-if-exists=false"
303305

304306
- name: run_precommit_and_push
305307
tags: [ "patch-run" ]
@@ -328,9 +330,10 @@ tasks:
328330
- func: clone
329331
- func: setup_building_host
330332
- func: quay_login
331-
- func: pipeline_agent
333+
- func: pipeline
332334
vars:
333-
FLAGS: "--all-agents --skip-if-exists=false"
335+
image_name: agent
336+
flags: "--parallel --all-agents --skip-if-exists=false"
334337

335338
- name: rebuild_currently_used_agents
336339
# this enables us to run this manually (patch) and rebuild current agent versions to verify
@@ -340,9 +343,10 @@ tasks:
340343
- func: clone
341344
- func: setup_building_host
342345
- func: quay_login
343-
- func: pipeline_agent
346+
- func: pipeline
344347
vars:
345-
FLAGS: "--current-agents --skip-if-exists=false"
348+
image_name: agent
349+
flags: "--parallel --current-agents --skip-if-exists=false"
346350

347351
- name: build_test_image
348352
commands:
@@ -352,7 +356,7 @@ tasks:
352356
- func: build_multi_cluster_binary
353357
- func: pipeline
354358
vars:
355-
IMAGE_NAME: meko-tests
359+
image_name: meko-tests
356360

357361
- name: build_test_image_arm
358362
commands:
@@ -362,8 +366,7 @@ tasks:
362366
- func: build_multi_cluster_binary
363367
- func: pipeline
364368
vars:
365-
IMAGE_NAME: meko-tests-arm64
366-
369+
image_name: meko-tests-arm64
367370

368371
- name: build_test_image_ibm
369372
commands:
@@ -387,76 +390,81 @@ tasks:
387390
- func: setup_building_host
388391
- func: pipeline
389392
vars:
390-
IMAGE_NAME: mco-tests
393+
image_name: mco-tests
391394

392395
- name: build_operator_ubi
393396
commands:
394397
- func: clone
395398
- func: setup_building_host
396399
- func: pipeline
397400
vars:
398-
IMAGE_NAME: operator
401+
image_name: operator
399402

400403
- name: build_operator_race_ubi
401404
commands:
402405
- func: clone
403406
- func: setup_building_host
404407
- func: pipeline
405408
vars:
406-
IMAGE_NAME: operator-race
409+
image_name: operator-race
407410

408411
- name: build_init_om_images_ubi
409412
commands:
410413
- func: clone
411414
- func: setup_building_host
412415
- func: pipeline
413416
vars:
414-
IMAGE_NAME: init-ops-manager
417+
image_name: init-ops-manager
415418

416419
- name: build_init_appdb_images_ubi
417420
commands:
418421
- func: clone
419422
- func: setup_building_host
420423
- func: pipeline
421424
vars:
422-
IMAGE_NAME: init-appdb
425+
image_name: init-appdb
423426

424427
- name: build_agent_images_ubi
425428
commands:
426429
- func: clone
427430
- func: setup_building_host
428431
- func: quay_login
429-
- func: pipeline_agent
432+
- func: pipeline
430433
vars:
431-
FLAGS: "--all-agents"
434+
image_name: agent
435+
flags: "--parallel --all-agents"
432436

433437
- name: build_init_database_image_ubi
434438
commands:
435439
- func: clone
436440
- func: setup_building_host
437441
- func: pipeline
438442
vars:
439-
IMAGE_NAME: init-database
443+
image_name: init-database
440444

441445
- name: build_database_image_ubi
442446
commands:
443447
- func: clone
444448
- func: setup_building_host
445449
- func: pipeline
446450
vars:
447-
IMAGE_NAME: database
451+
image_name: database
448452

449453
- name: build_readiness_probe_image
450454
commands:
451455
- func: clone
452456
- func: setup_building_host
453-
- func: pipeline_readiness_probe
457+
- func: pipeline
458+
vars:
459+
image_name: readiness-probe
454460

455461
- name: build_version_upgrade_hook_image
456462
commands:
457463
- func: clone
458464
- func: setup_building_host
459-
- func: pipeline_version_upgrade_hook
465+
- func: pipeline
466+
vars:
467+
image_name: upgrade-hook
460468

461469
- name: publish_helm_chart
462470
commands:
@@ -513,7 +521,9 @@ tasks:
513521
- func: clone
514522
- func: switch_context
515523
- func: setup_building_host
516-
- func: pipeline_ops_manager
524+
- func: pipeline
525+
vars:
526+
image_name: ops-manager
517527

518528
- name: publish_ops_manager
519529
commands:
@@ -522,9 +532,10 @@ tasks:
522532
- func: setup_building_host
523533
- func: quay_login
524534
- func: setup_docker_sbom
525-
- func: pipeline_ops_manager
535+
- func: pipeline
526536
vars:
527-
BUILD_SCENARIO: release
537+
image_name: ops-manager
538+
build_scenario: release
528539

529540
- name: prepare_and_upload_openshift_bundles_for_e2e
530541
commands:

‎scripts/release/pipeline.sh‎

Lines changed: 35 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,4 +4,38 @@ set -Eeou pipefail
44

55
source scripts/dev/set_env_context.sh
66

7-
scripts/dev/run_python.sh scripts/release/pipeline.py "${IMAGE_NAME}" --build-scenario "${BUILD_SCENARIO}" --version "${OPERATOR_VERSION}" ${FLAGS:+${FLAGS}}
7+
args=("${IMAGE_NAME}")
8+
args+=(--build-scenario "${BUILD_SCENARIO_OVERRIDE:-${BUILD_SCENARIO}}")
9+
10+
case ${IMAGE_NAME} in
11+
"agent")
12+
IMAGE_VERSION=""
13+
;;
14+
15+
"ops-manager")
16+
IMAGE_VERSION="${OM_VERSION}"
17+
;;
18+
19+
"readiness-probe")
20+
IMAGE_VERSION="${READINESS_PROBE_VERSION}"
21+
;;
22+
23+
"upgrade-hook")
24+
IMAGE_VERSION="${VERSION_UPGRADE_HOOK_VERSION}"
25+
;;
26+
27+
*)
28+
IMAGE_VERSION="${OPERATOR_VERSION}"
29+
;;
30+
esac
31+
32+
if [[ "${IMAGE_VERSION:-}" != "" ]]; then
33+
args+=(--version "${IMAGE_VERSION}")
34+
fi
35+
36+
if [[ "${FLAGS:-}" != "" ]]; then
37+
IFS=" " read -ra flags <<< "${FLAGS}"
38+
args+=("${flags[@]}")
39+
fi
40+
41+
scripts/dev/run_python.sh scripts/release/pipeline.py "${args[@]}"

‎scripts/release/pipeline_agent.sh‎

Lines changed: 0 additions & 16 deletions
This file was deleted.

0 commit comments

Comments
 (0)