Skip to content

Commit d8c7e3f

Browse files
cschreibcschreib
committed
Improve benchmark
1 parent fb1d526 commit d8c7e3f

File tree

6 files changed

+215
-141
lines changed

6 files changed

+215
-141
lines changed

‎README.md‎

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -122,27 +122,27 @@ Notes:
122122

123123
## Speed benchmarks
124124

125-
Labels are the same as in the comparison spreadsheet. The speed benchmarks were compiled with gcc 9.3.0 and libstdc++, with all optimizations turned on (except LTO), and run on a linux (5.1.0-89) machine with a Ryzen 5 2600 CPU. Speed is measured relative to `std::unique_ptr<T>` used as owner pointer, and `T*` used as observer pointer.
125+
Labels are the same as in the comparison spreadsheet. The speed benchmarks were compiled with gcc 9.3.0 and libstdc++, with all optimizations turned on (except LTO), and run on a linux (5.1.0-89) machine with a Ryzen 5 2600 CPU. Speed is measured relative to `std::unique_ptr<T>` used as owner pointer, and `T*` used as observer pointer, which should be the fastest possible implementation (but obviously the one with least safety).
126126

127-
You can run the benchmarks yourself, they are located in `tests/speed_benchmark.cpp`. The benchmark executable runs tests for three object types: `int`, `std::string`, and `std::array<int,65'536>`, to simulate objects of various allocation cost. The timings below are reported for `int`, which should be most relevant to highlight the overhead from the pointer itself. In real life scenarios, the actual measured overhead will be substantially lower, as actual business logic is likely to dominate the time budget.
127+
You can run the benchmarks yourself, they are located in `tests/speed_benchmark.cpp`. The benchmark executable runs tests for three object types: `int`, `float`, `std::string`, and `std::array<int,65'536>`, to simulate objects of various allocation cost. The timings below are the worst-case values measured across all object types, which should be most relevant to highlight the overhead from the pointer itself (and erases flukes from the benchmarking framework). In real life scenarios, the actual measured overhead will be substantially lower, as actual business logic is likely to dominate the time budget.
128128

129129
| Pointer | raw/unique | weak/shared | observer/obs_unique | observer/obs_sealed |
130130
|--------------------------|------------|-------------|---------------------|---------------------|
131-
| Create owner empty | 1 | 0.72 | 0.83 | 1 |
132-
| Create owner | 1 | 2.58 | 1.85 | N/A |
133-
| Create owner factory | 1 | 1.40 | 1.80 | 1.13 |
131+
| Create owner empty | 1 | 1.1 | 1.1 | 1.1 |
132+
| Create owner | 1 | 2.2 | 1.9 | N/A |
133+
| Create owner factory | 1 | 1.3 | 1.8 | 1.3 |
134134
| Dereference owner | 1 | 1 | 1 | 1 |
135-
| Create observer empty | 1 | 1 | 0.71 | 0.71 |
136-
| Create observer | 1 | 2.14 | 2.14 | 2.04 |
137-
| Create observer copy | 1 | 3.00 | 3.00 | 3.00 |
138-
| Dereference observer | 1 | 3.50 | 0.75 | 0.75 |
135+
| Create observer empty | 1 | 1.2 | 1.2 | 1.3 |
136+
| Create observer | 1 | 1.5 | 1.6 | 1.6 |
137+
| Create observer copy | 1 | 1.7 | 1.7 | 1.7 |
138+
| Dereference observer | 1 | 4.8 | 1.2 | 1.3 |
139139

140140
Detail of the benchmarks:
141-
- Create owner empty: default-construct an owner pointer (contains nullptr).
141+
- Create owner empty: default-construct an owner pointer (to nullptr).
142142
- Create owner: construct an owner pointer by taking ownership of an object (for `oup::observer_sealed_ptr`, this is using `oup::make_observable_sealed()`).
143143
- Create owner factory: construct an owner pointer using `std::make_*` or `oup::make_*` factory functions.
144144
- Dereference owner: get a reference to the underlying owned object from an owner pointer.
145-
- Create observer empty: default-construct an observer pointer (contains nullptr).
145+
- Create observer empty: default-construct an observer pointer (to nullptr).
146146
- Create observer: construct an observer pointer from an owner pointer.
147147
- Create observer copy: construct a new observer pointer from another observer pointer.
148148
- Dereference observer: get a reference to the underlying object from an observer pointer.

‎tests/CMakeLists.txt‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -80,5 +80,6 @@ target_link_libraries(oup_size_benchmark PRIVATE oup::oup)
8080

8181
add_executable(oup_speed_benchmark
8282
${PROJECT_SOURCE_DIR}/tests/speed_benchmark.cpp
83-
${PROJECT_SOURCE_DIR}/tests/speed_benchmark_utility.cpp)
83+
${PROJECT_SOURCE_DIR}/tests/speed_benchmark_utility.cpp
84+
${PROJECT_SOURCE_DIR}/tests/speed_benchmark_utility2.cpp)
8485
target_link_libraries(oup_speed_benchmark PRIVATE oup::oup)

‎tests/speed_benchmark.cpp‎

Lines changed: 31 additions & 129 deletions
Original file line numberDiff line numberDiff line change
@@ -1,150 +1,49 @@
1-
#include <memory>
2-
#include <iostream>
3-
#include <chrono>
4-
#include <array>
5-
#include <string>
6-
#include <oup/observable_unique_ptr.hpp>
7-
8-
// External functions, the compiler cannot see through. Prevents optimisations.
9-
template<typename T>
10-
void use_object(T&) noexcept;
11-
12-
template<typename T>
13-
struct pointer_traits;
14-
15-
template<typename T>
16-
struct pointer_traits<std::unique_ptr<T>> {
17-
using element_type = T;
18-
using ptr_type = std::unique_ptr<T>;
19-
using weak_type = T*;
20-
21-
static ptr_type make_ptr() noexcept { return ptr_type(new element_type); }
22-
static ptr_type make_ptr_factory() noexcept { return std::make_unique<element_type>(); }
23-
static weak_type make_weak(ptr_type& p) noexcept { return p.get(); }
24-
template<typename F>
25-
static void deref_weak(weak_type& p, F&& func) noexcept { return func(*p); }
26-
};
27-
28-
template<typename T>
29-
struct pointer_traits<std::shared_ptr<T>> {
30-
using element_type = T;
31-
using ptr_type = std::shared_ptr<T>;
32-
using weak_type = std::weak_ptr<T>;
33-
34-
static ptr_type make_ptr() noexcept { return ptr_type(new element_type); }
35-
static ptr_type make_ptr_factory() noexcept { return std::make_shared<element_type>(); }
36-
static weak_type make_weak(ptr_type& p) noexcept { return weak_type(p); }
37-
template<typename F>
38-
static void deref_weak(weak_type& p, F&& func) noexcept { if (auto s = p.lock()) func(*s); }
39-
};
40-
41-
template<typename T>
42-
struct pointer_traits<oup::observable_unique_ptr<T>> {
43-
using element_type = T;
44-
using ptr_type = oup::observable_unique_ptr<T>;
45-
using weak_type = oup::observer_ptr<T>;
46-
47-
static ptr_type make_ptr() noexcept { return ptr_type(new element_type); }
48-
static ptr_type make_ptr_factory() noexcept { return oup::make_observable_unique<element_type>(); }
49-
static weak_type make_weak(ptr_type& p) noexcept { return weak_type(p); }
50-
template<typename F>
51-
static void deref_weak(weak_type& p, F&& func) noexcept { return func(*p); }
52-
};
53-
54-
template<typename T>
55-
struct pointer_traits<oup::observable_sealed_ptr<T>> {
56-
using element_type = T;
57-
using ptr_type = oup::observable_sealed_ptr<T>;
58-
using weak_type = oup::observer_ptr<T>;
59-
60-
static ptr_type make_ptr() noexcept { return oup::make_observable_sealed<element_type>(); }
61-
static ptr_type make_ptr_factory() noexcept { return oup::make_observable_sealed<element_type>(); }
62-
static weak_type make_weak(ptr_type& p) noexcept { return weak_type(p); }
63-
template<typename F>
64-
static void deref_weak(weak_type& p, F&& func) noexcept { return func(*p); }
65-
};
66-
67-
template<typename T>
68-
struct benchmark {
69-
using traits = pointer_traits<T>;
70-
using element_type = typename traits::element_type;
71-
using owner_type = typename traits::ptr_type;
72-
using weak_type = typename traits::weak_type;
73-
74-
owner_type owner;
75-
weak_type weak;
76-
77-
benchmark() : owner(traits::make_ptr()), weak(traits::make_weak(owner)) {}
78-
79-
void construct_destruct_owner_empty() {
80-
auto p = owner_type{};
81-
use_object(p);
82-
}
83-
84-
void construct_destruct_owner() {
85-
auto p = traits::make_ptr();
86-
use_object(p);
87-
}
88-
89-
void construct_destruct_owner_factory() {
90-
auto p = traits::make_ptr_factory();
91-
use_object(p);
92-
}
93-
94-
void construct_destruct_weak_empty() {
95-
auto p = weak_type{};
96-
use_object(p);
97-
}
98-
99-
void construct_destruct_weak() {
100-
auto wp = traits::make_weak(owner);
101-
use_object(wp);
102-
}
103-
104-
void dereference_owner() {
105-
use_object(*owner);
106-
}
107-
108-
void dereference_weak() {
109-
traits::deref_weak(weak, [](auto& o) { use_object(o); });
110-
}
111-
void construct_destruct_weak_copy() {
112-
auto wp = weak;
113-
use_object(wp);
114-
}
115-
};
116-
117-
using timer = std::chrono::high_resolution_clock;
1+
#include "speed_benchmark_common.hpp"
2+
#include <cmath>
1183

1194
template<typename B, typename F>
120-
double run_benchmark_for(F&& func) {
5+
auto run_benchmark_for(F&& func) {
1216
B bench{};
1227

123-
auto prev = timer::now();
1248
double elapsed = 0.0;
9+
double elapsed_square = 0.0;
12510
double count = 0.0;
126-
constexpr std::size_t num_iter = 10'000;
11+
double attempts = 0.0;
12+
constexpr std::size_t num_iter = 1'000'000;
13+
14+
while (elapsed*num_iter < 1.0) {
15+
auto prev = timer::now();
12716

128-
while (elapsed < 1.0) {
12917
for (std::size_t i = 0; i < num_iter; ++i) {
13018
func(bench);
13119
}
13220

13321
auto now = timer::now();
134-
elapsed += std::chrono::duration_cast<std::chrono::duration<double>>(now - prev).count();
135-
count += static_cast<double>(num_iter);
136-
std::swap(now, prev);
22+
23+
double spent = std::chrono::duration_cast<std::chrono::duration<double>>(now - prev).count()/num_iter;
24+
elapsed += spent;
25+
elapsed_square += spent*spent;
26+
attempts += 1.0;
13727
}
13828

139-
return elapsed/count;
29+
double stddev = std::sqrt(elapsed_square/attempts - (elapsed/attempts)*(elapsed/attempts))/std::sqrt(attempts);
30+
31+
return std::make_pair(elapsed/attempts, stddev);
14032
}
14133

14234
template<typename B, typename F>
14335
auto run_benchmark(F&& func) {
14436
using ref_type = benchmark<std::unique_ptr<typename B::element_type>>;
145-
double result = run_benchmark_for<B>(func);
146-
double result_ref = run_benchmark_for<ref_type>(func);
147-
return std::make_pair(result, result/result_ref);
37+
38+
auto result = run_benchmark_for<B>(func);
39+
auto result_ref = run_benchmark_for<ref_type>(func);
40+
41+
double ratio = result.first/result_ref.first;
42+
double rel_err = result.second/result.first;
43+
double rel_err_ref = result_ref.second/result_ref.first;
44+
double ratio_stddev = std::sqrt(rel_err*rel_err + rel_err_ref*rel_err_ref)*ratio;
45+
46+
return std::make_pair(result, std::make_pair(ratio, ratio_stddev));
14847
}
14948

15049
template<typename T>
@@ -161,7 +60,9 @@ void do_benchmarks_for_ptr(const char* type_name, const char* ptr_name) {
16160
auto dereference_weak = run_benchmark<B>([](auto& b) { return b.dereference_weak(); });
16261

16362
std::cout << ptr_name << "<" << type_name << ">:" << std::endl;
164-
#define report(which) std::cout << " - " << #which << ": " << which.first*1e6 << "us (x" << which.second << ")" << std::endl
63+
#define report(which) std::cout << " - " << #which << ": " << \
64+
which.first.first*1e6 << " +/- " << which.first.second*1e6 << "us " << \
65+
"(x" << which.second.first << " +/- " << which.second.second << ")" << std::endl
16566

16667
report(construct_destruct_owner_empty);
16768
report(construct_destruct_owner);
@@ -185,6 +86,7 @@ void do_benchmarks(const char* type_name) {
18586

18687
int main() {
18788
do_benchmarks<int>("int");
89+
do_benchmarks<float>("float");
18890
do_benchmarks<std::string>("string");
18991
do_benchmarks<std::array<int,65'536>>("big_array");
19092
return 0;

‎tests/speed_benchmark_common.hpp‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
#include <memory>
2+
#include <iostream>
3+
#include <chrono>
4+
#include <array>
5+
#include <string>
6+
#include <oup/observable_unique_ptr.hpp>
7+
8+
// External functions, the compiler cannot see through. Prevents optimisations.
9+
template<typename T>
10+
void use_object(T&) noexcept;
11+
12+
template<typename T>
13+
struct pointer_traits;
14+
15+
template<typename T>
16+
struct pointer_traits<std::unique_ptr<T>> {
17+
using element_type = T;
18+
using ptr_type = std::unique_ptr<T>;
19+
using weak_type = T*;
20+
21+
static ptr_type make_ptr() noexcept { return ptr_type(new element_type); }
22+
static ptr_type make_ptr_factory() noexcept { return std::make_unique<element_type>(); }
23+
static weak_type make_weak(ptr_type& p) noexcept { return p.get(); }
24+
template<typename F>
25+
static void deref_weak(weak_type& p, F&& func) noexcept { return func(*p); }
26+
};
27+
28+
template<typename T>
29+
struct pointer_traits<std::shared_ptr<T>> {
30+
using element_type = T;
31+
using ptr_type = std::shared_ptr<T>;
32+
using weak_type = std::weak_ptr<T>;
33+
34+
static ptr_type make_ptr() noexcept { return ptr_type(new element_type); }
35+
static ptr_type make_ptr_factory() noexcept { return std::make_shared<element_type>(); }
36+
static weak_type make_weak(ptr_type& p) noexcept { return weak_type(p); }
37+
template<typename F>
38+
static void deref_weak(weak_type& p, F&& func) noexcept { if (auto s = p.lock()) func(*s); }
39+
};
40+
41+
template<typename T>
42+
struct pointer_traits<oup::observable_unique_ptr<T>> {
43+
using element_type = T;
44+
using ptr_type = oup::observable_unique_ptr<T>;
45+
using weak_type = oup::observer_ptr<T>;
46+
47+
static ptr_type make_ptr() noexcept { return ptr_type(new element_type); }
48+
static ptr_type make_ptr_factory() noexcept { return oup::make_observable_unique<element_type>(); }
49+
static weak_type make_weak(ptr_type& p) noexcept { return weak_type(p); }
50+
template<typename F>
51+
static void deref_weak(weak_type& p, F&& func) noexcept { return func(*p); }
52+
};
53+
54+
template<typename T>
55+
struct pointer_traits<oup::observable_sealed_ptr<T>> {
56+
using element_type = T;
57+
using ptr_type = oup::observable_sealed_ptr<T>;
58+
using weak_type = oup::observer_ptr<T>;
59+
60+
static ptr_type make_ptr() noexcept { return oup::make_observable_sealed<element_type>(); }
61+
static ptr_type make_ptr_factory() noexcept { return oup::make_observable_sealed<element_type>(); }
62+
static weak_type make_weak(ptr_type& p) noexcept { return weak_type(p); }
63+
template<typename F>
64+
static void deref_weak(weak_type& p, F&& func) noexcept { return func(*p); }
65+
};
66+
67+
template<typename T>
68+
struct benchmark {
69+
using traits = pointer_traits<T>;
70+
using element_type = typename traits::element_type;
71+
using owner_type = typename traits::ptr_type;
72+
using weak_type = typename traits::weak_type;
73+
74+
owner_type owner;
75+
weak_type weak;
76+
77+
benchmark() : owner(traits::make_ptr()), weak(traits::make_weak(owner)) {}
78+
79+
void construct_destruct_owner_empty();
80+
81+
void construct_destruct_owner();
82+
83+
void construct_destruct_owner_factory();
84+
85+
void construct_destruct_weak_empty();
86+
87+
void construct_destruct_weak();
88+
89+
void construct_destruct_weak_copy();
90+
91+
void dereference_owner();
92+
93+
void dereference_weak();
94+
};
95+
96+
using timer = std::chrono::high_resolution_clock;

‎tests/speed_benchmark_utility.cpp‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,33 +7,41 @@ template<typename T>
77
void use_object(T&) noexcept {}
88

99
template void use_object<int>(int&) noexcept;
10+
template void use_object<float>(float&) noexcept;
1011
template void use_object<std::string>(std::string&) noexcept;
1112
template void use_object<std::array<int,65'536>>(std::array<int,65'536>&) noexcept;
1213

1314
template void use_object<int*>(int*&) noexcept;
15+
template void use_object<float*>(float*&) noexcept;
1416
template void use_object<std::string*>(std::string*&) noexcept;
1517
template void use_object<std::array<int,65'536>*>(std::array<int,65'536>*&) noexcept;
1618

1719
template void use_object<std::unique_ptr<int>>(std::unique_ptr<int>&) noexcept;
20+
template void use_object<std::unique_ptr<float>>(std::unique_ptr<float>&) noexcept;
1821
template void use_object<std::unique_ptr<std::string>>(std::unique_ptr<std::string>&) noexcept;
1922
template void use_object<std::unique_ptr<std::array<int,65'536>>>(std::unique_ptr<std::array<int,65'536>>&) noexcept;
2023

2124
template void use_object<std::shared_ptr<int>>(std::shared_ptr<int>&) noexcept;
25+
template void use_object<std::shared_ptr<float>>(std::shared_ptr<float>&) noexcept;
2226
template void use_object<std::shared_ptr<std::string>>(std::shared_ptr<std::string>&) noexcept;
2327
template void use_object<std::shared_ptr<std::array<int,65'536>>>(std::shared_ptr<std::array<int,65'536>>&) noexcept;
2428

2529
template void use_object<std::weak_ptr<int>>(std::weak_ptr<int>&) noexcept;
30+
template void use_object<std::weak_ptr<float>>(std::weak_ptr<float>&) noexcept;
2631
template void use_object<std::weak_ptr<std::string>>(std::weak_ptr<std::string>&) noexcept;
2732
template void use_object<std::weak_ptr<std::array<int,65'536>>>(std::weak_ptr<std::array<int,65'536>>&) noexcept;
2833

2934
template void use_object<oup::observable_unique_ptr<int>>(oup::observable_unique_ptr<int>&) noexcept;
35+
template void use_object<oup::observable_unique_ptr<float>>(oup::observable_unique_ptr<float>&) noexcept;
3036
template void use_object<oup::observable_unique_ptr<std::string>>(oup::observable_unique_ptr<std::string>&) noexcept;
3137
template void use_object<oup::observable_unique_ptr<std::array<int,65'536>>>(oup::observable_unique_ptr<std::array<int,65'536>>&) noexcept;
3238

3339
template void use_object<oup::observable_sealed_ptr<int>>(oup::observable_sealed_ptr<int>&) noexcept;
40+
template void use_object<oup::observable_sealed_ptr<float>>(oup::observable_sealed_ptr<float>&) noexcept;
3441
template void use_object<oup::observable_sealed_ptr<std::string>>(oup::observable_sealed_ptr<std::string>&) noexcept;
3542
template void use_object<oup::observable_sealed_ptr<std::array<int,65'536>>>(oup::observable_sealed_ptr<std::array<int,65'536>>&) noexcept;
3643

3744
template void use_object<oup::observer_ptr<int>>(oup::observer_ptr<int>&) noexcept;
45+
template void use_object<oup::observer_ptr<float>>(oup::observer_ptr<float>&) noexcept;
3846
template void use_object<oup::observer_ptr<std::string>>(oup::observer_ptr<std::string>&) noexcept;
3947
template void use_object<oup::observer_ptr<std::array<int,65'536>>>(oup::observer_ptr<std::array<int,65'536>>&) noexcept;

0 commit comments

Comments
 (0)