Skip to content

Commit 976020c

Browse files
vr009vr009
committed
test: add benchmark tests
Added benchmarks of large Select and Replace. Added a new target in Makefile for running benchmark tests. Added a new space in config.lua for large Select tests. Added a new target in Makefile for measuring performance degradation between current changes and master. Added a new line in gitignore for ignoring artifacts from bench target. Added a new step for running benchmark tests in ci. Added description to the CONTRIBUTING.md for how to run benchmark tests. Closes #122
1 parent 9fb381c commit 976020c

File tree

6 files changed

+231
-0
lines changed

6 files changed

+231
-0
lines changed

‎.github/workflows/testing.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,3 +63,6 @@ jobs:
6363
COVERALLS_TOKEN: ${{ secrets.GITHUB_TOKEN }}
6464
run: |
6565
make coveralls
66+
67+
- name: Check workability of benchmark tests
68+
run: make bench-deps bench DURATION=1x COUNT=1

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,3 +3,4 @@
33
.idea/
44
work_dir*
55
.rocks
6+
bench*

‎CONTRIBUTING.md‎

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,113 @@ and run it with next flags:
4545
golangci-lint run -E gofmt -D errcheck
4646
```
4747

48+
## Benchmarking
49+
50+
### Quick start
51+
52+
To run all benchmark tests from the current branch run:
53+
54+
```bash
55+
make bench
56+
```
57+
58+
To measure performance difference between master and the current branch run:
59+
60+
```bash
61+
make bench-diff
62+
```
63+
64+
Note: `benchstat` should be in `PATH`. If it is not set, call:
65+
66+
```bash
67+
export PATH="/home/${USER}/go/bin:${PATH}"
68+
```
69+
70+
or
71+
72+
```bash
73+
export PATH="${HOME}/go/bin:${PATH}"
74+
```
75+
76+
### Customize benchmarking
77+
78+
Before running benchmark or measuring performance degradation, install benchmark dependencies:
79+
```bash
80+
make bench-deps BENCH_PATH=custom_path
81+
```
82+
83+
Use the variable `BENCH_PATH` to specify the path of benchmark artifacts.
84+
It is set to `bench` by default.
85+
86+
To run benchmark tests, call:
87+
```bash
88+
make bench DURATION=5s COUNT=7 BENCH_PATH=custom_path TEST_PATH=.
89+
```
90+
91+
Use the variable `DURATION` to set the duration of perf tests. That variable is mapped on
92+
testing [flag](https://pkg.go.dev/cmd/go#hdr-Testing_flags) `-benchtime` for gotest.
93+
It may take the values in seconds (e.g, `5s`) or count of iterations (e.g, `1000x`).
94+
It is set to `3s` by default.
95+
96+
Use the variable `COUNT` to control the count of benchmark runs for each test.
97+
It is set to `5` by default. That variable is mapped on testing flag `-count`.
98+
Use higher values if the benchmark numbers aren't stable.
99+
100+
Use the variable `TEST_PATH` to set the directory of test files.
101+
It is set to `./...` by default, so it runs all the Benchmark tests in the project.
102+
103+
To measure performance degradation after changes in code, run:
104+
```bash
105+
make bench-diff BENCH_PATH=custom_path
106+
```
107+
108+
Note: the variable `BENCH_PATH` is not purposed to be used with absolute paths.
109+
110+
## Recommendations for how to achieve stable results
111+
112+
Before any judgments, verify whether results are stable on given host and how large the noise. Run `make bench-diff` without changes and look on the report. Several times.
113+
114+
There are suggestions how to achieve best results:
115+
116+
* Close all background applications, especially web browser. Look at `top` (`htop`, `atop`, ...) and if something bubbles there, close it.
117+
* Disable cron daemon.
118+
* Disable TurboBoost and set fixed frequency.
119+
* If you're using `intel_pstate` frequency driver (it is usually default):
120+
121+
Disable TurboBoost:
122+
123+
```shell
124+
$ echo 0 > /sys/devices/system/cpu/intel_pstate/no_turbo
125+
```
126+
127+
Set fixed frequency: not sure it is possible.
128+
129+
* If you're using `acpi-cpufreq` driver:
130+
131+
Ensure you actually don't use intel_pstate:
132+
133+
```shell
134+
$ grep -o 'intel_pstate=\w\+' /proc/cmdline
135+
intel_pstate=disable
136+
$ cpupower -c all frequency-info | grep driver:
137+
driver: acpi-cpufreq
138+
<...>
139+
```
140+
141+
Disable TurboBoost:
142+
143+
```shell
144+
$ echo 0 > /sys/devices/system/cpu/cpufreq/boost
145+
```
146+
147+
Set fixed frequency:
148+
149+
```shell
150+
$ cpupower -c all frequency-set -g userspace
151+
$ cpupower -c all frequency-set -f 1.80GHz # adjust for your CPU
152+
```
153+
154+
48155
## Code review checklist
49156

50157
- Public API contains functions, variables, constants that are needed from

‎Makefile‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,18 @@
11
SHELL := /bin/bash
22
COVERAGE_FILE := coverage.out
3+
MAKEFILE_PATH := $(abspath $(lastword $(MAKEFILE_LIST)))
4+
PROJECT_DIR := $(patsubst %/,%,$(dir $(MAKEFILE_PATH)))
5+
DURATION ?= 3s
6+
COUNT ?= 5
7+
BENCH_PATH ?= bench-dir
8+
TEST_PATH ?= ${PROJECT_DIR}/...
9+
BENCH_FILE := ${PROJECT_DIR}/${BENCH_PATH}/bench.txt
10+
REFERENCE_FILE := ${PROJECT_DIR}/${BENCH_PATH}/reference.txt
11+
BENCH_FILES := ${REFERENCE_FILE} ${BENCH_FILE}
12+
BENCH_REFERENCE_REPO := ${BENCH_PATH}/go-tarantool
13+
BENCH_OPTIONS := -bench=. -run=^Benchmark -benchmem -benchtime=${DURATION} -count=${COUNT}
14+
GO_TARANTOOL_URL := https://github.com/tarantool/go-tarantool
15+
GO_TARANTOOL_DIR := ${PROJECT_DIR}/${BENCH_PATH}/go-tarantool
316

417
.PHONY: clean
518
clean:
@@ -55,3 +68,33 @@ coverage:
5568
coveralls: coverage
5669
go get github.com/mattn/goveralls
5770
goveralls -coverprofile=$(COVERAGE_FILE) -service=github
71+
72+
.PHONY: bench-deps
73+
${BENCH_PATH} bench-deps:
74+
@echo "Installing benchstat tool"
75+
rm -rf ${BENCH_PATH}
76+
mkdir ${BENCH_PATH}
77+
go clean -testcache
78+
cd ${BENCH_PATH} && git clone https://go.googlesource.com/perf && cd perf && go install ./cmd/benchstat
79+
rm -rf ${BENCH_PATH}/perf
80+
81+
.PHONY: bench
82+
${BENCH_FILE} bench: ${BENCH_PATH}
83+
@echo "Running benchmark tests from the current branch"
84+
go test ${TEST_PATH} ${BENCH_OPTIONS} 2>&1 \
85+
| tee ${BENCH_FILE}
86+
benchstat ${BENCH_FILE}
87+
88+
${GO_TARANTOOL_DIR}:
89+
@echo "Cloning the repository into ${GO_TARANTOOL_DIR}"
90+
[ ! -e ${GO_TARANTOOL_DIR} ] && git clone --depth=1 ${GO_TARANTOOL_URL} ${GO_TARANTOOL_DIR}
91+
92+
${REFERENCE_FILE}: ${GO_TARANTOOL_DIR}
93+
@echo "Running benchmark tests from master for using results in bench-diff target"
94+
cd ${GO_TARANTOOL_DIR} && git pull && go test ./... ${BENCH_OPTIONS} 2>&1 \
95+
| tee ${REFERENCE_FILE}
96+
97+
bench-diff: ${BENCH_FILES}
98+
@echo "Comparing performance between master and the current branch"
99+
@echo "'old' is a version in master branch, 'new' is a version in a current branch"
100+
benchstat ${BENCH_FILES} | grep -v pkg:

‎config.lua‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,31 @@ box.once("init", function()
4040
})
4141
st:truncate()
4242

43+
local s2 = box.schema.space.create('test_perf', {
44+
id = 520,
45+
temporary = true,
46+
if_not_exists = true,
47+
field_count = 3,
48+
format = {
49+
{name = "id", type = "unsigned"},
50+
{name = "name", type = "string"},
51+
{name = "arr1", type = "array"},
52+
},
53+
})
54+
s2:create_index('primary', {type = 'tree', unique = true, parts = {1, 'unsigned'}, if_not_exists = true})
55+
s2:create_index('secondary', {id = 5, type = 'tree', unique = false, parts = {2, 'string'}, if_not_exists = true})
56+
local arr_data = {}
57+
for i = 1,100 do
58+
arr_data[i] = i
59+
end
60+
for i = 1,1000 do
61+
s2:insert{
62+
i,
63+
'test_name',
64+
arr_data,
65+
}
66+
end
67+
4368
--box.schema.user.grant('guest', 'read,write,execute', 'universe')
4469
box.schema.func.create('box.info')
4570
box.schema.func.create('simple_incr')
@@ -49,6 +74,7 @@ box.once("init", function()
4974
box.schema.user.grant('test', 'execute', 'universe')
5075
box.schema.user.grant('test', 'read,write', 'space', 'test')
5176
box.schema.user.grant('test', 'read,write', 'space', 'schematest')
77+
box.schema.user.grant('test', 'read,write', 'space', 'test_perf')
5278
end)
5379

5480
local function func_name()

‎tarantool_test.go‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -362,6 +362,57 @@ func BenchmarkClientParallelMassiveUntyped(b *testing.B) {
362362
close(limit)
363363
}
364364

365+
func BenchmarkClientReplaceParallel(b *testing.B) {
366+
conn, err := Connect(server, opts)
367+
if err != nil {
368+
b.Errorf("No connection available")
369+
return
370+
}
371+
defer conn.Close()
372+
spaceNo = 520
373+
374+
rSpaceNo, _, err := conn.Schema.ResolveSpaceIndex("test_perf", "secondary")
375+
if err != nil {
376+
b.Fatalf("Space is not resolved: %s", err.Error())
377+
}
378+
379+
b.ResetTimer()
380+
b.RunParallel(func(pb *testing.PB) {
381+
for pb.Next() {
382+
_, err := conn.Replace(rSpaceNo, []interface{}{uint(1), "hello", []interface{}{}})
383+
if err != nil {
384+
b.Error(err)
385+
}
386+
}
387+
})
388+
}
389+
390+
func BenchmarkClientLargeSelectParallel(b *testing.B) {
391+
conn, err := Connect(server, opts)
392+
if err != nil {
393+
b.Errorf("No connection available")
394+
return
395+
}
396+
defer conn.Close()
397+
398+
schema := conn.Schema
399+
rSpaceNo, rIndexNo, err := schema.ResolveSpaceIndex("test_perf", "secondary")
400+
if err != nil {
401+
b.Fatalf("symbolic space and index params not resolved")
402+
}
403+
404+
offset, limit := uint32(0), uint32(1000)
405+
b.ResetTimer()
406+
b.RunParallel(func(pb *testing.PB) {
407+
for pb.Next() {
408+
_, err := conn.Select(rSpaceNo, rIndexNo, offset, limit, IterEq, []interface{}{"test_name"})
409+
if err != nil {
410+
b.Fatal(err)
411+
}
412+
}
413+
})
414+
}
415+
365416
///////////////////
366417

367418
func TestClient(t *testing.T) {

0 commit comments

Comments
 (0)