Skip to content

Commit 69faa43

Browse files
committed
Fix benchmark instructions: differentiate tasty-bench & gauge options. [skip ci]
1 parent 6b04711 commit 69faa43

File tree

1 file changed

+55
-2
lines changed

1 file changed

+55
-2
lines changed

benchmark/README.md

Lines changed: 55 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,55 @@
1+
# Benchmark library choice
2+
3+
You may choose between two libraries to perform the benchmarks:
4+
5+
- `tasty-bench`: default
6+
- `gauge`: alternative, requires cabal flag `use-gauge`.
7+
18
# Comparing benchmarks for regression
29

310
The following commands are tested with `cabal` version 3.0.
411

12+
## `tasty-bench`
13+
14+
Run the benchmarks for the baseline code i.e. without the changes:
15+
16+
```
17+
# Remove any old benchmark results file first
18+
$ rm results.csv
19+
$ cabal run bench -- --csv=old.csv
20+
```
21+
22+
It will collect the benchmark results in `old.csv` file.
23+
24+
If you want more accurate benchmark results you can add `--stdev 1`
25+
(default: 5).
26+
27+
You may generate a SVG chart by adding `--svg chart.svg`.
28+
29+
Modify the code and then run:
30+
31+
```
32+
$ cabal run bench -- --csv=new.csv --baseline=old.csv
33+
```
34+
35+
It will collect the new benchmark results in `new.csv` file.
36+
37+
To generate a benchmark comparison in CSV between old and new changes from the
38+
benchmark results:
39+
40+
```
41+
# Source: see “Comparison against baseline” in tasty-bench documentation.
42+
awk 'BEGIN{FS=",";OFS=",";print "Name,Old,New,Ratio"}FNR==1{next}FNR==NR{a[$1]=$2;next}{print $1,a[$1],$2,$2/a[$1];gs+=log($2/a[$1]);gc++}END{print "Geometric mean,,",exp(gs/gc)}' old.csv new.csv
43+
```
44+
45+
## `gauge`
46+
547
Run the benchmarks for the baseline code i.e. without the changes:
648

749
```
850
# Remove any old benchmark results file first
951
$ rm results.csv
10-
$ cabal run bench -- --csvraw=results.csv --quick
52+
$ cabal run bench --flag use-gauge -- --csvraw=results.csv --quick
1153
```
1254

1355
It will collect the benchmark results in `results.csv` file.
@@ -51,11 +93,22 @@ $ export LIBRARY_PATH=/usr/lib/:/opt/local/lib
5193
$ cabal bench --extra-lib-dirs=/usr/local/opt/icu4c/lib --extra-include-dirs=/usr/local/opt/icu4c/include
5294
```
5395

96+
## `tasty-bench`
97+
98+
Remove any old `results.csv` and run benchmarks with `has-icu` flag enabled:
99+
100+
```
101+
$ rm results.csv
102+
$ cabal run bench --flag has-icu -- --csv=results.csv
103+
```
104+
105+
## `gauge`
106+
54107
Remove any old `results.csv` and run benchmarks with `has-icu` flag enabled:
55108

56109
```
57110
$ rm results.csv
58-
$ cabal run bench --flag has-icu -- --csvraw=results.csv --quick
111+
$ cabal run bench -f use-gauge -f has-icu -- --csvraw=results.csv --quick
59112
```
60113

61114
The following command will now show the comparison between `text-icu` and

0 commit comments

Comments
 (0)