Skip to content

Commit fe81867

Browse files
lsierantlsierant
authored
Switch context by test (#204)
# Summary This PR helps in switching to a proper context/variant by specifying a test mark or full task's evergreen url. It solves a source of friction in our day-to-day workflows, which is finding which context to switch to in order to run a particular test. And by being able to switch context by specifying evergreen task's url we're a step closer to having automation that reruns a failed script locally. ## Current workflow To find a context to run the test locally we often start from two places: 1. Starting from Evergreen UI (easier): look at the evergreen variant in the evergreen UI, copy it and run `make switch context=<copied variant>` or use interactive fzf-powered `make switch` (without args), paste or type the variant and press enter. 2. Starting from an e2e test file (harder): find a `@pytest.mark` in the file, copy it, go to evergreen.yml, find a task group containing that file, copy its name, then find a variant where that task group is used, copy variant name, run `make switch context=<copied variant>`... ## Proposed workflow `make switcht` is a new tool which is a supercharged `make switch`. It's named similarly (typing `make switch` is kind of a muscle memory now) and works in two modes: 1. With explicit test mark/url on `test=` argument: - `make switcht test=<test>`, where `<test>` is a pytest mark (e.g. `e2e_replica_set`) - it will list all the variants that this test is running in to be picked with `fzf` - `make switcht test=<task url>`, where `<task url>` is a full url to the evergreen test - it will switch directly to the variant from that test 2. Without specifying test argument, which works in a similar interactive way like `make switch`, but lists first all test files with their marks to fuzzy-find the test, which then is used to list pick its variants: `make switcht` ## Requirements It requires: - [fzf](https://github.com/junegunn/fzf) - for the interactive fuzzy finder (we already use it for `make switch`) - [ripgrep](https://github.com/BurntSushi/ripgrep) - to *quickly* list all the pytest marks in all e2e test files - Having evg credentials in env vars, e.g. in your `scripts/dev/contexts/private-context` ``` EVERGREEN_USER=$(cat ~/.evergreen.yml | yq .user) EVERGREEN_API_KEY=$(cat ~/.evergreen.yml | yq .api_key) ``` ## Changes Majority of the implementation is in two places: - `scripts/dev/switch_context_by_test.sh` - it's mostly for `fzf`, finding pytest marks using `rg`. It calls scripts/python/find_test_variants.py - scripts/python/find_test_variants.py - a new python script that lists all the variants of a pytest's mark (`--task-name`) or a variant from a full evg task url (`--task-url`) - introduced scripts/python directory, to limit proliferation of python scripts in random directories. Having them in one place makes it easier to import/run them as modules - refactored some evergreen API helpers into scripts/python/evergreen_api.py - some minor, random changes improving debuggability of bash scripts ## Proof of Work Here are some examples for the most common use cases: **With explicit test= parameter** - Switch to one of test's context by pytest mark: `make switcht test=e2e_replica_set` ![switcht with mark](https://github.com/user-attachments/assets/f05b7cd3-d674-4cd4-9dc2-034eb9f37e23) - Switch to context by test's url: `make switcht test='https://spruce.mongodb.com/task/[...]/logs?execution=0'` ![switcht with url](https://github.com/user-attachments/assets/2add88f1-9a45-4b47-bbda-3d75b86ca00e) **Without any parameters (fully interactive)** - Switch to context by fuzzy-searching a test first: `make switcht` ![switcht by fuzzy search](https://github.com/user-attachments/assets/991cedcc-8663-4aa4-9f8b-9cc9f4af516a) - Switch to context by pasting full file path into test picker: `make switcht` ![switcht by pasting full filepath](https://github.com/user-attachments/assets/860e344c-cbe5-4cf7-b48a-71907af49ab8) - Switch to context by pasting full url into fzf picker: `make switcht` ![switcht by pasting url](https://github.com/user-attachments/assets/b7c0f5c4-30d2-4e9f-abf1-8a7e2197f09a) Afterwards, if the context needs to be regenerated again just run `make switch` (without t) + enter - it will remember last switched context and be selected first. ## Checklist - [ ] Have you linked a jira ticket and/or is the ticket in the title? - [ ] Have you checked whether your jira ticket required DOCSP changes? - [ ] Have you checked for release_note changes? ## Reminder (Please remove this when merging) - Please try to Approve or Reject Changes the PR, keep PRs in review as short as possible - Our Short Guide for PRs: [Link](https://docs.google.com/document/d/1T93KUtdvONq43vfTfUt8l92uo4e4SEEvFbIEKOxGr44/edit?tab=t.0) - Remember the following Communication Standards - use comment prefixes for clarity: * **blocking**: Must be addressed before approval. * **follow-up**: Can be addressed in a later PR or ticket. * **q**: Clarifying question. * **nit**: Non-blocking suggestions. * **note**: Side-note, non-actionable. Example: Praise * --> no prefix is considered a question
1 parent 789b071 commit fe81867

File tree

13 files changed

+300
-27
lines changed

13 files changed

+300
-27
lines changed

‎Makefile‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,9 @@ precommit:
6262
switch:
6363
@ scripts/dev/switch_context.sh $(context) $(additional_override)
6464

65+
switcht:
66+
@ scripts/dev/switch_context_by_test.sh $(test)
67+
6568
# builds the Operator binary file and docker image and pushes it to the remote registry if using a remote registry. Deploys it to
6669
# k8s cluster
6770
operator: configure-operator build-and-push-operator-image

‎scripts/__init__.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
# Makes 'scripts' a Python package.

‎scripts/dev/contexts/root-context‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -116,3 +116,8 @@ export MDB_SEARCH_COMMUNITY_VERSION
116116

117117
export MDB_SEARCH_COMMUNITY_NAME="mongodb-search-community"
118118
export MDB_SEARCH_COMMUNITY_REPO_URL="quay.io/mongodb"
119+
120+
121+
if [[ ${MDB_BASH_DEBUG:-0} == 1 ]]; then
122+
export PS4='+(${BASH_SOURCE}:${LINENO})[^$?]: ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'
123+
fi

‎scripts/dev/launch_e2e.sh‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22

33
set -Eeou pipefail
44

5+
test "${MDB_BASH_DEBUG:-0}" -eq 1 && set -x
56

67
# The script launches e2e test. Note, that the Operator and necessary resources are deployed
78
# inside the test
@@ -49,5 +50,3 @@ else
4950
fi
5051

5152
title "E2e test ${test} is finished"
52-
53-

‎scripts/dev/set_env_context.sh‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,14 @@
11
#!/usr/bin/env bash
22

33
set -Eeou pipefail
4+
test "${MDB_BASH_DEBUG:-0}" -eq 1 && set -x
45

56
# shellcheck disable=1091
67
source scripts/funcs/errors
78

89
script_name=$(readlink -f "${BASH_SOURCE[0]}")
910
script_dir=$(dirname "${script_name}")
10-
context_file="${script_dir}/../../.generated/context.export.env"
11+
context_file="$(realpath "${script_dir}/../../.generated/context.export.env")"
1112

1213
if [[ ! -f ${context_file} ]]; then
1314
fatal "File ${context_file} not found! Make sure to follow this guide to get started: https://wiki.corp.mongodb.com/display/MMS/Setting+up+local+development+and+E2E+testing#SettinguplocaldevelopmentandE2Etesting-GettingStartedGuide(VariantSwitching)"

‎scripts/dev/switch_context_by_test.sh‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
#!/usr/bin/env bash
2+
3+
set -Eeou pipefail
4+
test "${MDB_BASH_DEBUG:-0}" -eq 1 && set -x
5+
6+
usage() {
7+
echo "Switch context by passing the test (evergreen task name or full evergreen task URL)."
8+
echo "If there is more than one variant running given test, then fzf picker is used."
9+
echo "Usage: $0 <test>"
10+
echo " <test> is a task name from .evergreen.yml (e.g. 'e2e_search_community_basic') or a full Evergreen task URL."
11+
}
12+
13+
source scripts/funcs/errors
14+
15+
list_pytest_marks() {
16+
rg -g '*.py' -o --no-line-number --no-heading --replace '$1' -m 1 \
17+
'@(?:pytest\.)?mark\.(e2e_[a-zA-Z0-9_]+)' \
18+
docker/mongodb-kubernetes-tests
19+
}
20+
21+
pick_test_by_file_mark_or_task_url() {
22+
if ! test_list="$(list_pytest_marks | sort -u)"; then
23+
echo "Couldn't list pytest marks."
24+
echo "${test_list}"
25+
return 1
26+
fi
27+
28+
test=$(fzf --print-query --header-first --with-nth '{2}: {1}' -d ':' --accept-nth 2 \
29+
--header "Select file or task to find contexts where its used. You can paste full task's evergreen url here" <<< "${test_list}") || true
30+
if [[ -z ${test} ]]; then
31+
echo "Aborted selecting test"
32+
return 1
33+
fi
34+
35+
# test may contain one or two lines (file:mark or just mark/url)
36+
number_of_selected_lines=$(wc -l <<< "${test}")
37+
if [[ ${number_of_selected_lines} -eq 2 ]]; then
38+
test="$(tail -n 1 <<< "${test}")"
39+
elif [[ ${number_of_selected_lines} -gt 2 ]]; then
40+
echo "Too many lines selected: ${test}"
41+
return 1
42+
fi
43+
44+
echo "${test}"
45+
}
46+
47+
main() {
48+
test="${1:-}"
49+
50+
if [[ -z ${test} ]]; then
51+
test=$(pick_test_by_file_mark_or_task_url)
52+
echo "Selected test: ${test}"
53+
fi
54+
55+
if [[ "${test}" = *spruce.mongodb.com* ]]; then
56+
find_variant_arg="--task-url"
57+
else
58+
find_variant_arg="--task-name"
59+
fi
60+
61+
if ! contexts=$(scripts/evergreen/run_python.sh scripts/python/find_test_variants.py "${find_variant_arg}" "${test}"); then
62+
echo "Couldn't find any test contexts running test: ${test}"
63+
echo "${contexts}"
64+
exit 1
65+
fi
66+
67+
echo "Found contexts that are running test: ${test}"
68+
echo "${contexts}"
69+
70+
selected_context="${contexts}"
71+
if [[ $(wc -l <<< "${contexts}") -gt 1 ]]; then
72+
if ! selected_context=$(fzf --header "${test} runs in multiple variants/contexts. Select one to switch context into." --header-first --layout=reverse <<< "${contexts}"); then
73+
echo "Aborted selecting context"
74+
exit 1
75+
fi
76+
fi
77+
78+
scripts/dev/switch_context.sh "${selected_context}"
79+
}
80+
81+
main "$@"

‎scripts/evergreen/flakiness-report.py‎ renamed to ‎scripts/evergreen/flakiness_report.py‎

Lines changed: 5 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,9 @@
1-
import json
2-
import os
31
import sys
42

53
import requests
64

5+
from scripts.python.evergreen_api import get_evergreen_auth_headers
6+
77
EVERGREEN_API = "https://evergreen.mongodb.com/api"
88

99

@@ -18,16 +18,13 @@ def print_usage():
1818

1919

2020
def get_variants_with_retried_tasks() -> dict[str, list[dict]]:
21-
evg_user = os.environ.get("EVERGREEN_USER", "")
22-
api_key = os.environ.get("API_KEY", "")
23-
24-
if len(sys.argv) != 2 or evg_user == "" or api_key == "":
21+
if len(sys.argv) != 2:
2522
print_usage()
26-
exit(1)
23+
raise RuntimeError("Exactly one argument (patch version number) must be provided")
2724

2825
version = sys.argv[1]
26+
headers = get_evergreen_auth_headers()
2927

30-
headers = {"Api-User": evg_user, "Api-Key": api_key}
3128
print("Fetching build variants...", file=sys.stderr)
3229
build_ids = requests.get(url=f"{EVERGREEN_API}/rest/v2/versions/{version}", headers=headers).json()
3330
build_statuses = [build_status for build_status in build_ids["build_variants_status"]]

‎scripts/funcs/checks‎

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,5 @@
11
#!/usr/bin/env bash
22

3-
pushd "${PWD}" > /dev/null || return
4-
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
5-
cd "${DIR}" || return
6-
source errors
7-
popd > /dev/null || return
8-
93
check_env_var() {
104
local var_name="$1"
115
local msg="$2"

‎scripts/funcs/kubernetes‎

Lines changed: 3 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -2,16 +2,9 @@
22

33
set -Eeou pipefail
44

5-
pushd "${PWD}" > /dev/null || return
6-
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
7-
cd "${DIR}" || return
8-
# shellcheck source=scripts/funcs/checks
9-
source checks
10-
# shellcheck source=scripts/funcs/errors
11-
source errors
12-
# shellcheck source=scripts/funcs/printing
13-
source printing
14-
popd > /dev/null || return
5+
source scripts/funcs/checks
6+
source scripts/funcs/errors
7+
source scripts/funcs/printing
158

169
ensure_namespace() {
1710
local namespace="${1}"

‎scripts/python/__init__.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
# Makes 'scripts.python' a Python package.

0 commit comments

Comments
 (0)