Enforce release builds for benchmarks and simplify command interface (#741)

* Initial plan

Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>

* Enforce release builds for benchmarks in Justfile and source code

Co-authored-by: jsturtevant <648372+jsturtevant@users.noreply.github.com>
Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>

* Remove compile-time error for benchmarks to allow profiling with debug symbols

Co-authored-by: ludfjig <4257730+ludfjig@users.noreply.github.com>
Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>

* Remove unnecessary target parameter from bench commands

- Remove target parameter from bench and bench-ci commands since they only accept release builds
- Update GitHub workflows to remove target parameter usage
- Simplify command signatures based on @syntactically's feedback

Co-authored-by: syntactically <168595099+syntactically@users.noreply.github.com>
Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>

* Update documentation to reflect that debug builds are now supported for profiling

Co-authored-by: ludfjig <4257730+ludfjig@users.noreply.github.com>
Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>

---------

Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: jsturtevant <648372+jsturtevant@users.noreply.github.com>
Co-authored-by: ludfjig <4257730+ludfjig@users.noreply.github.com>
Co-authored-by: syntactically <168595099+syntactically@users.noreply.github.com>

Author: 4 people

.github/workflows/Benchmarks.yml

@@ -57,7 +57,7 @@ jobs:
           GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
 
       - name: Run Benchmarks
-        run: just bench-ci main release ${{ matrix.hypervisor == 'mshv' && 'mshv2' || ''}}
+        run: just bench-ci main ${{ matrix.hypervisor == 'mshv' && 'mshv2' || ''}}
 
       - uses: actions/upload-artifact@v4
         with:

.github/workflows/dep_rust.yml

@@ -186,5 +186,5 @@ jobs:
 
       - name: Run benchmarks
         run: |
-          just bench-ci main ${{ matrix.config }} ${{ matrix.hypervisor == 'mshv' && 'mshv2' || ''}}
+          just bench-ci main ${{ matrix.hypervisor == 'mshv' && 'mshv2' || ''}}
         if: ${{ matrix.config == 'release' }}

Justfile

@@ -326,11 +326,13 @@ bench-download os hypervisor cpu tag="":
     tar -zxvf target/benchmarks_{{ os }}_{{ hypervisor }}_{{ cpu }}.tar.gz -C target/criterion/ --strip-components=1
 
 # Warning: compares to and then OVERWRITES the given baseline
-bench-ci baseline target=default-target features="":
-    cargo bench --profile={{ if target == "debug" { "dev" } else { target } }} {{ if features =="" {''} else { "--features " + features } }} -- --verbose --save-baseline {{ baseline }}
+bench-ci baseline features="":
+    @# Benchmarks are always run with release builds for meaningful results
+    cargo bench --profile=release {{ if features =="" {''} else { "--features " + features } }} -- --verbose --save-baseline {{ baseline }}
 
-bench target=default-target features="":
-    cargo bench --profile={{ if target == "debug" { "dev" } else { target } }} {{ if features =="" {''} else { "--features " + features } }} -- --verbose
+bench features="":
+    @# Benchmarks are always run with release builds for meaningful results
+    cargo bench --profile=release {{ if features =="" {''} else { "--features " + features } }} -- --verbose
 
 ###############
 ### FUZZING ###

docs/benchmarking-hyperlight.md

@@ -72,4 +72,6 @@ Found 1 outliers among 100 measurements (1.00%)
 
 ## Running benchmarks locally
 
-Use `just bench [debug/release]` parameter to run benchmarks. Comparing local benchmarks results to github-saved benchmarks doesn't make much sense, since you'd be using different hardware, but you can use `just bench-download os hypervisor [tag] ` to download and extract the GitHub release benchmarks to the correct place folder. You can then run `just bench-ci main` to compare to (and overwrite) the previous release benchmarks. Note that `main` is the name of the baselines stored in GitHub.
+Use `just bench` to run benchmarks with release builds (the only supported configuration). Comparing local benchmark results to github-saved benchmarks doesn't make much sense, since you'd be using different hardware, but you can use `just bench-download os hypervisor [tag] ` to download and extract the GitHub release benchmarks to the correct place folder. You can then run `just bench-ci main` to compare to (and overwrite) the previous release benchmarks. Note that `main` is the name of the baselines stored in GitHub.
+
+**Important**: The `just bench` command uses release builds by default to ensure meaningful performance measurements. For profiling purposes, you can compile benchmarks with debug symbols by running `cargo bench` directly.