chore: task reexecution #4443
+397
−198
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2,30 +2,34 @@ name: 'C-Chain Re-Execution Benchmark' | |
| description: 'Run C-Chain re-execution benchmark' | ||
|
|
||
| inputs: | ||
| task: | ||
| description: 'Task name to execute from Taskfile.yml. Leave empty to use custom inputs below.' | ||
| default: '' | ||
| # Custom inputs (alternative to task-based approach) | ||
| config: | ||
| description: 'The config to pass to the VM for the benchmark. See BenchmarkReexecuteRange for details.' | ||
| default: '' | ||
| start-block: | ||
| description: 'The start block for the benchmark.' | ||
| default: '' | ||
| end-block: | ||
| description: 'The end block for the benchmark.' | ||
| default: '' | ||
| block-dir-src: | ||
| description: 'The source block directory. Supports S3 directory/zip and local directories.' | ||
| default: '' | ||
| current-state-dir-src: | ||
| description: 'The current state directory. Supports S3 directory/zip and local directories.' | ||
| default: '' | ||
| runner_name: | ||
| description: 'The name of the runner to use and include in the Golang Benchmark name.' | ||
| required: true | ||
| aws-role: | ||
| description: 'AWS role to assume for S3 access.' | ||
| required: true | ||
| aws-region: | ||
| description: 'AWS region to use for S3 access.' | ||
| default: 'us-east-2' | ||
| aws-role-duration-seconds: | ||
| description: 'The duration of the AWS role to assume for S3 access.' | ||
| required: true | ||
|
|
@@ -56,54 +60,126 @@ inputs: | |
| push-github-action-benchmark: | ||
| description: 'Whether to push the benchmark result to GitHub.' | ||
| required: true | ||
| push-post-state: | ||
| description: 'S3 destination to copy the current-state directory after completing re-execution. If empty, this will be skipped.' | ||
| default: '' | ||
| # The following inputs need never be provided by the caller. They | ||
| # default to context values that the action's steps are unable to | ||
| # access directly. | ||
| repository-owner: | ||
| default: ${{ github.repository_owner }} | ||
| repository-name: | ||
| default: ${{ github.event.repository.name }} | ||
| workflow: | ||
| default: ${{ github.workflow }} | ||
| run-id: | ||
| default: ${{ github.run_id }} | ||
| run-number: | ||
| default: ${{ github.run_number }} | ||
| run-attempt: | ||
| default: ${{ github.run_attempt }} | ||
| job: | ||
| default: ${{ github.job }} | ||
|
|
||
| runs: | ||
| using: composite | ||
| steps: | ||
| - uses: cachix/install-nix-action@02a151ada4993995686f9ed4f1be7cfbb229e56f #v31 | ||
| with: | ||
| github_access_token: ${{ inputs.github-token }} | ||
| - run: echo "dependencies installed" | ||
| shell: nix develop --command bash {0} | ||
| # Cache Go modules (architecture-independent) | ||
| - uses: actions/cache@v4 | ||
| id: go-mod-cache | ||
| with: | ||
| path: ~/go/pkg/mod | ||
| key: ${{ runner.os }}-go-mod-${{ hashFiles('go.sum') }} | ||
| restore-keys: ${{ runner.os }}-go-mod- | ||
| # Cache Go build cache (architecture-specific) | ||
| - uses: actions/cache@v4 | ||
| with: | ||
| path: ~/.cache/go-build | ||
| key: ${{ runner.os }}-${{ runner.arch }}-go-build-${{ hashFiles('go.sum') }} | ||
| restore-keys: ${{ runner.os }}-${{ runner.arch }}-go-build- | ||
| # Download modules only on cache miss | ||
| - run: go mod download | ||
| if: steps.go-mod-cache.outputs.cache-hit != 'true' | ||
| shell: nix develop --command bash -x {0} | ||
| - name: Notify of metrics availability | ||
| if: inputs.prometheus-username != '' | ||
| shell: bash | ||
| run: | | ||
| metrics_url=$($GITHUB_ACTION_PATH/output-metrics-url.sh) | ||
| echo "Grafana: ${metrics_url}" | ||
| echo "🔗 [View Grafana Dashboard](${metrics_url})" >> "$GITHUB_STEP_SUMMARY" | ||
| env: | ||
| GRAFANA_URL: https://grafana-poc.avax-dev.network/d/Gl1I20mnk/c-chain?orgId=1&refresh=10s&var-filter=is_ephemeral_node%7C%3D%7Cfalse&var-filter=gh_repo%7C%3D%7C${{ inputs.repository_owner }}%2F${{ inputs.repository_name }}&var-filter=gh_run_id%7C%3D%7C${{ inputs.run_id }}&var-filter=gh_run_attempt%7C%3D%7C${{ inputs.run_attempt }} | ||
| GH_JOB_ID: ${{ inputs.job }} | ||
| - name: Warn that collection of metrics and logs will not be performed | ||
| if: inputs.prometheus-username == '' | ||
| shell: bash | ||
| run: echo "::warning::Monitoring credentials not found. Skipping collector start. Is the PR from a fork branch?" | ||
| - name: Configure AWS Credentials | ||
| uses: aws-actions/configure-aws-credentials@v4 | ||
| with: | ||
| role-to-assume: ${{ inputs.aws-role }} | ||
| aws-region: ${{ inputs.aws-region }} | ||
| role-duration-seconds: ${{ inputs.aws-role-duration-seconds }} | ||
| - name: Validate inputs | ||
| shell: bash | ||
| run: | | ||
| if [[ -z "${{ inputs.task }}" ]]; then | ||
| # Granular mode - validate required inputs | ||
| missing=() | ||
| [[ -z "${{ inputs.block-dir-src }}" ]] && missing+=("block-dir-src") | ||
| [[ -z "${{ inputs.current-state-dir-src }}" ]] && missing+=("current-state-dir-src") | ||
| [[ -z "${{ inputs.start-block }}" ]] && missing+=("start-block") | ||
| [[ -z "${{ inputs.end-block }}" ]] && missing+=("end-block") | ||
|
|
||
| if [[ ${#missing[@]} -gt 0 ]]; then | ||
| echo "::error::When 'task' is empty, the following inputs are required: ${missing[*]}" | ||
| exit 1 | ||
| fi | ||
| fi | ||
| - name: Set task env | ||
| shell: bash | ||
| run: | | ||
| TIMESTAMP=$(date '+%Y%m%d-%H%M%S') | ||
| echo "EXECUTION_DATA_DIR=/tmp/reexecution-data-${TIMESTAMP}" >> "$GITHUB_ENV" | ||
| echo "BENCHMARK_OUTPUT_FILE=${GITHUB_WORKSPACE}/benchmark-output.txt" >> "$GITHUB_ENV" | ||
| - name: Run C-Chain Re-execution Benchmark | ||
| shell: nix develop --impure --command bash -x {0} | ||
| run: | | ||
| if [[ -n "${{ inputs.task }}" ]]; then | ||
| # Task-based approach | ||
| ./scripts/run_task.sh ${{ inputs.task }} \ | ||
| BENCHMARK_OUTPUT_FILE="${{ env.BENCHMARK_OUTPUT_FILE }}" \ | ||
| EXECUTION_DATA_DIR="${{ env.EXECUTION_DATA_DIR }}" | ||
| else | ||
| # Granular approach | ||
| ./scripts/run_task.sh reexecute-cchain-range-with-copied-data \ | ||
| CONFIG=${{ inputs.config }} \ | ||
| EXECUTION_DATA_DIR=${{ env.EXECUTION_DATA_DIR }} \ | ||
| BLOCK_DIR_SRC=${{ inputs.block-dir-src }} \ | ||
| CURRENT_STATE_DIR_SRC=${{ inputs.current-state-dir-src }} \ | ||
| START_BLOCK=${{ inputs.start-block }} \ | ||
| END_BLOCK=${{ inputs.end-block }} \ | ||
| BENCHMARK_OUTPUT_FILE="${{ env.BENCHMARK_OUTPUT_FILE }}" | ||
| fi | ||
| env: | ||
|
Collaborator
There was a problem hiding this comment. Why were some env vars removed in this change? It seems like metrics server enabled and labels were dropped here
Contributor
Author
There was a problem hiding this comment. In current master labels were never set, not even in the parent action for example we can see labels are empty on this run. Regarding metrics server enabled we consolidated it making it implicitly controlled by whether the collector is enabled. The |
||
| RUNNER_NAME: ${{ inputs.runner_name }} | ||
| METRICS_COLLECTOR_ENABLED: ${{ inputs.prometheus-username != '' }} | ||
| PROMETHEUS_URL: ${{ inputs.prometheus-url }} | ||
| PROMETHEUS_PUSH_URL: ${{ inputs.prometheus-push-url }} | ||
| PROMETHEUS_USERNAME: ${{ inputs.prometheus-username }} | ||
| PROMETHEUS_PASSWORD: ${{ inputs.prometheus-password }} | ||
| GH_REPO: ${{ inputs.repository_owner }}/${{ inputs.repository_name }} | ||
| GH_WORKFLOW: ${{ inputs.workflow }} | ||
| GH_RUN_ID: ${{ inputs.run_id }} | ||
| GH_RUN_NUMBER: ${{ inputs.run_number }} | ||
| GH_RUN_ATTEMPT: ${{ inputs.run_attempt }} | ||
| GH_JOB_ID: ${{ inputs.job }} | ||
| - name: Compare Benchmark Results | ||
| uses: benchmark-action/github-action-benchmark@v1 | ||
| with: | ||
|
|
@@ -112,8 +188,10 @@ runs: | |
| summary-always: true | ||
| github-token: ${{ inputs.github-token }} | ||
| auto-push: ${{ inputs.push-github-action-benchmark }} | ||
| - name: Push Post-State to S3 | ||
| if: inputs.push-post-state != '' | ||
| shell: nix develop --impure --command bash -x {0} | ||
| run: | | ||
| ./scripts/run_task.sh export-dir-to-s3 \ | ||
| SRC=${{ env.EXECUTION_DATA_DIR }}/current-state/ \ | ||
| DST=${{ inputs.push-post-state }} | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| # WARNING: This file is a duplication of: | ||
| # - .github/actions/run-monitored-tmpnet-cmd/output-metrics-url.sh (source of truth) | ||
| # Changes must be made to BOTH files. | ||
|
|
||
| set -euo pipefail | ||
|
|
||
| # Timestamps are in seconds | ||
| from_timestamp="$(date '+%s')" | ||
| monitoring_period=900 # 15 minutes | ||
| to_timestamp="$((from_timestamp + monitoring_period))" | ||
|
|
||
| # Grafana expects microseconds, so pad timestamps with 3 zeros | ||
| metrics_url="${GRAFANA_URL}&var-filter=gh_job_id%7C%3D%7C${GH_JOB_ID}&from=${from_timestamp}000&to=${to_timestamp}000" | ||
|
|
||
| # Optionally ensure that the link displays metrics only for the shared | ||
| # network rather than mixing it with the results for private networks. | ||
| if [[ -n "${FILTER_BY_OWNER:-}" ]]; then | ||
| metrics_url="${metrics_url}&var-filter=network_owner%7C%3D%7C${FILTER_BY_OWNER}" | ||
| fi | ||
|
|
||
| echo "${metrics_url}" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -6,34 +6,39 @@ on: | |
| inputs: | ||
| config: | ||
| description: 'The config to pass to the VM for the benchmark. See BenchmarkReexecuteRange for details.' | ||
| default: '' | ||
| start-block: | ||
| description: 'The start block for the benchmark.' | ||
| default: '' | ||
| end-block: | ||
| description: 'The end block for the benchmark.' | ||
| default: '' | ||
| block-dir-src: | ||
| description: 'The source block directory. Supports S3 directory/zip and local directories.' | ||
| default: '' | ||
| current-state-dir-src: | ||
| description: 'The current state directory. Supports S3 directory/zip and local directories.' | ||
| default: '' | ||
| task: | ||
Elvis339 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| description: 'Taskfile task to execute (e.g., c-chain-reexecution-hashdb-101-250k)' | ||
| default: '' | ||
| runner-preset: | ||
| description: 'Select a predefined runner (ignored if custom-runner is provided)' | ||
| type: choice | ||
| options: | ||
| - ubuntu-latest | ||
| - avalanche-avalanchego-runner-2ti | ||
| - avago-runner-m6i-4xlarge-ebs-fast | ||
| - avago-runner-i4i-4xlarge-local-ssd | ||
| default: ubuntu-latest | ||
| custom-runner: | ||
| description: 'Custom runner name (overrides runner-preset if provided)' | ||
| default: '' | ||
| push-post-state: | ||
| description: 'S3 location to push post-execution state directory. Skips this step if left unpopulated.' | ||
| default: '' | ||
| timeout-minutes: | ||
| description: 'Timeout in minutes for the job.' | ||
| default: 30 | ||
|
|
||
| # Disabled because scheduled trigger is empty. To enable, uncomment and add at least one vector to the schedule | ||
|
|
@@ -53,15 +58,20 @@ jobs: | |
| shell: bash -x {0} | ||
| run: | | ||
| if [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then | ||
| # Use custom-runner if provided (after trimming whitespace), otherwise use runner-preset | ||
| CUSTOM_RUNNER=$(echo "${{ github.event.inputs.custom-runner }}" | xargs) | ||
|
|
||
| if [[ -n "$CUSTOM_RUNNER" ]]; then | ||
| RUNNER="$CUSTOM_RUNNER" | ||
| else | ||
| RUNNER="${{ github.event.inputs.runner-preset }}" | ||
| fi | ||
|
|
||
| { | ||
| echo "matrix<<EOF" | ||
| printf '{ "include": [{ "task": "%s", "runner": "%s", "timeout-minutes": %s }] }\n' \ | ||
| "${{ github.event.inputs.task }}" \ | ||
| "$RUNNER" \ | ||
| "${{ github.event.inputs.timeout-minutes }}" | ||
| echo EOF | ||
| } >> "$GITHUB_OUTPUT" | ||
|
|
@@ -100,11 +110,12 @@ jobs: | |
| - name: Run C-Chain Re-Execution Benchmark | ||
| uses: ./.github/actions/c-chain-reexecution-benchmark | ||
| with: | ||
| task: ${{ matrix.task }} | ||
|
Collaborator
There was a problem hiding this comment. Since we use
Contributor
Author
There was a problem hiding this comment. If you mean to update JSON file with dynamic inputs (runner, start, end block...) then no, you'd have to create a task. What's the need for this when we support dynamic dispatch assuming I understood the question. |
||
| config: ${{ inputs.config }} | ||
| start-block: ${{ inputs.start-block }} | ||
| end-block: ${{ inputs.end-block }} | ||
| block-dir-src: ${{ inputs.block-dir-src }} | ||
| current-state-dir-src: ${{ inputs.current-state-dir-src }} | ||
| prometheus-url: ${{ secrets.PROMETHEUS_URL || '' }} | ||
| prometheus-push-url: ${{ secrets.PROMETHEUS_PUSH_URL || '' }} | ||
| prometheus-username: ${{ secrets.PROMETHEUS_USERNAME || '' }} | ||
|
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.