|
| 1 | +# Benchmark library choice |
| 2 | + |
| 3 | +You may choose between two libraries to perform the benchmarks: |
| 4 | + |
| 5 | +- `tasty-bench`: default |
| 6 | +- `gauge`: alternative, requires cabal flag `use-gauge`. |
| 7 | + |
1 | 8 | # Comparing benchmarks for regression |
2 | 9 |
|
3 | 10 | The following commands are tested with `cabal` version 3.0. |
4 | 11 |
|
| 12 | +## `tasty-bench` |
| 13 | + |
| 14 | +Run the benchmarks for the baseline code i.e. without the changes: |
| 15 | + |
| 16 | +``` |
| 17 | +# Remove any old benchmark results file first |
| 18 | +$ rm results.csv |
| 19 | +$ cabal run bench -- --csv=old.csv |
| 20 | +``` |
| 21 | + |
| 22 | +It will collect the benchmark results in `old.csv` file. |
| 23 | + |
| 24 | +If you want more accurate benchmark results you can add `--stdev 1` |
| 25 | +(default: 5). |
| 26 | + |
| 27 | +You may generate a SVG chart by adding `--svg chart.svg`. |
| 28 | + |
| 29 | +Modify the code and then run: |
| 30 | + |
| 31 | +``` |
| 32 | +$ cabal run bench -- --csv=new.csv --baseline=old.csv |
| 33 | +``` |
| 34 | + |
| 35 | +It will collect the new benchmark results in `new.csv` file. |
| 36 | + |
| 37 | +To generate a benchmark comparison in CSV between old and new changes from the |
| 38 | +benchmark results: |
| 39 | + |
| 40 | +``` |
| 41 | +# Source: see “Comparison against baseline” in tasty-bench documentation. |
| 42 | +awk 'BEGIN{FS=",";OFS=",";print "Name,Old,New,Ratio"}FNR==1{next}FNR==NR{a[$1]=$2;next}{print $1,a[$1],$2,$2/a[$1];gs+=log($2/a[$1]);gc++}END{print "Geometric mean,,",exp(gs/gc)}' old.csv new.csv |
| 43 | +``` |
| 44 | + |
| 45 | +## `gauge` |
| 46 | + |
5 | 47 | Run the benchmarks for the baseline code i.e. without the changes: |
6 | 48 |
|
7 | 49 | ``` |
8 | 50 | # Remove any old benchmark results file first |
9 | 51 | $ rm results.csv |
10 | | -$ cabal run bench -- --csvraw=results.csv --quick |
| 52 | +$ cabal run bench --flag use-gauge -- --csvraw=results.csv --quick |
11 | 53 | ``` |
12 | 54 |
|
13 | 55 | It will collect the benchmark results in `results.csv` file. |
@@ -51,11 +93,22 @@ $ export LIBRARY_PATH=/usr/lib/:/opt/local/lib |
51 | 93 | $ cabal bench --extra-lib-dirs=/usr/local/opt/icu4c/lib --extra-include-dirs=/usr/local/opt/icu4c/include |
52 | 94 | ``` |
53 | 95 |
|
| 96 | +## `tasty-bench` |
| 97 | + |
| 98 | +Remove any old `results.csv` and run benchmarks with `has-icu` flag enabled: |
| 99 | + |
| 100 | +``` |
| 101 | +$ rm results.csv |
| 102 | +$ cabal run bench --flag has-icu -- --csv=results.csv |
| 103 | +``` |
| 104 | + |
| 105 | +## `gauge` |
| 106 | + |
54 | 107 | Remove any old `results.csv` and run benchmarks with `has-icu` flag enabled: |
55 | 108 |
|
56 | 109 | ``` |
57 | 110 | $ rm results.csv |
58 | | -$ cabal run bench --flag has-icu -- --csvraw=results.csv --quick |
| 111 | +$ cabal run bench -f use-gauge -f has-icu -- --csvraw=results.csv --quick |
59 | 112 | ``` |
60 | 113 |
|
61 | 114 | The following command will now show the comparison between `text-icu` and |
|
0 commit comments