Skip to content

Commit 802fbf6

Browse files
gmilegmile
authored
Update insertion benchmark (#144)
* Simplify instructions for running benchmark MIX_ENV=bench doesn't exist, and postgrex and myxql are installed for MIX_ENV=dev, so we can simplify this * Bump mysql version from 5.7 to 8 5.7 won't start on arm64 architecture * Handle decoding of %Time{} struct
1 parent eab3893 commit 802fbf6

File tree

7 files changed

+582
-509
lines changed

7 files changed

+582
-509
lines changed

‎bench/README.md‎

Lines changed: 3 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -18,14 +18,8 @@ need PostgreSQL and MySQL up and running.
1818

1919
To run the benchmarks tests just type in the console:
2020

21-
```
22-
# POSIX-compatible shells
23-
$ MIX_ENV=bench mix run bench/bench_helper.exs
24-
```
25-
26-
```
27-
# other shells
28-
$ env MIX_ENV=bench mix run bench/bench_helper.exs
21+
```sh
22+
mix run bench/bench_helper.exs
2923
```
3024

3125
Benchmarks are inside the `scripts/` directory and are divided into two
@@ -44,6 +38,6 @@ script instead of `bench/bench_helper.exs`.
4438
The easiest way to setup mysql and postgresql for the benchmarks is via Docker. Run the following commands to get an instance of each running.
4539

4640
```
47-
docker run -p 3306:3306 -e MYSQL_ALLOW_EMPTY_PASSWORD=yes mysql:5.7
41+
docker run -p 3306:3306 -e MYSQL_ALLOW_EMPTY_PASSWORD=yes mysql:8
4842
docker run -p 5432:5432 -e POSTGRES_PASSWORD=postgres postgres:13.2
4943
```

‎bench/results/all.md‎

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,119 @@
1+
Benchmark
2+
3+
Benchmark run from 2024-05-04 10:50:11.956493Z UTC
4+
5+
## System
6+
7+
Benchmark suite executing on the following system:
8+
9+
<table style="width: 1%">
10+
<tr>
11+
<th style="width: 1%; white-space: nowrap">Operating System</th>
12+
<td>macOS</td>
13+
</tr><tr>
14+
<th style="white-space: nowrap">CPU Information</th>
15+
<td style="white-space: nowrap">Apple M3 Max</td>
16+
</tr><tr>
17+
<th style="white-space: nowrap">Number of Available Cores</th>
18+
<td style="white-space: nowrap">16</td>
19+
</tr><tr>
20+
<th style="white-space: nowrap">Available Memory</th>
21+
<td style="white-space: nowrap">128 GB</td>
22+
</tr><tr>
23+
<th style="white-space: nowrap">Elixir Version</th>
24+
<td style="white-space: nowrap">1.16.2</td>
25+
</tr><tr>
26+
<th style="white-space: nowrap">Erlang Version</th>
27+
<td style="white-space: nowrap">26.2.4</td>
28+
</tr>
29+
</table>
30+
31+
## Configuration
32+
33+
Benchmark suite executing with the following configuration:
34+
35+
<table style="width: 1%">
36+
<tr>
37+
<th style="width: 1%">:time</th>
38+
<td style="white-space: nowrap">10 s</td>
39+
</tr><tr>
40+
<th>:parallel</th>
41+
<td style="white-space: nowrap">1</td>
42+
</tr><tr>
43+
<th>:warmup</th>
44+
<td style="white-space: nowrap">2 s</td>
45+
</tr>
46+
</table>
47+
48+
## Statistics
49+
50+
51+
52+
Run Time
53+
54+
<table style="width: 1%">
55+
<tr>
56+
<th>Name</th>
57+
<th style="text-align: right">IPS</th>
58+
<th style="text-align: right">Average</th>
59+
<th style="text-align: right">Devitation</th>
60+
<th style="text-align: right">Median</th>
61+
<th style="text-align: right">99th&nbsp;%</th>
62+
</tr>
63+
64+
<tr>
65+
<td style="white-space: nowrap">MyXQL Repo.all/2</td>
66+
<td style="white-space: nowrap; text-align: right">639.68</td>
67+
<td style="white-space: nowrap; text-align: right">1.56 ms</td>
68+
<td style="white-space: nowrap; text-align: right">&plusmn;12.01%</td>
69+
<td style="white-space: nowrap; text-align: right">1.54 ms</td>
70+
<td style="white-space: nowrap; text-align: right">2.20 ms</td>
71+
</tr>
72+
73+
<tr>
74+
<td style="white-space: nowrap">SQLite3 Repo.all/2</td>
75+
<td style="white-space: nowrap; text-align: right">465.99</td>
76+
<td style="white-space: nowrap; text-align: right">2.15 ms</td>
77+
<td style="white-space: nowrap; text-align: right">&plusmn;4.82%</td>
78+
<td style="white-space: nowrap; text-align: right">2.13 ms</td>
79+
<td style="white-space: nowrap; text-align: right">2.43 ms</td>
80+
</tr>
81+
82+
<tr>
83+
<td style="white-space: nowrap">Pg Repo.all/2</td>
84+
<td style="white-space: nowrap; text-align: right">214.16</td>
85+
<td style="white-space: nowrap; text-align: right">4.67 ms</td>
86+
<td style="white-space: nowrap; text-align: right">&plusmn;12.05%</td>
87+
<td style="white-space: nowrap; text-align: right">4.66 ms</td>
88+
<td style="white-space: nowrap; text-align: right">5.86 ms</td>
89+
</tr>
90+
91+
</table>
92+
93+
94+
Run Time Comparison
95+
96+
<table style="width: 1%">
97+
<tr>
98+
<th>Name</th>
99+
<th style="text-align: right">IPS</th>
100+
<th style="text-align: right">Slower</th>
101+
<tr>
102+
<td style="white-space: nowrap">MyXQL Repo.all/2</td>
103+
<td style="white-space: nowrap;text-align: right">639.68</td>
104+
<td>&nbsp;</td>
105+
</tr>
106+
107+
<tr>
108+
<td style="white-space: nowrap">SQLite3 Repo.all/2</td>
109+
<td style="white-space: nowrap; text-align: right">465.99</td>
110+
<td style="white-space: nowrap; text-align: right">1.37x</td>
111+
</tr>
112+
113+
<tr>
114+
<td style="white-space: nowrap">Pg Repo.all/2</td>
115+
<td style="white-space: nowrap; text-align: right">214.16</td>
116+
<td style="white-space: nowrap; text-align: right">2.99x</td>
117+
</tr>
118+
119+
</table>

‎bench/results/insert.md‎

Lines changed: 50 additions & 59 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,6 @@
1+
Benchmark
12

2-
# Benchmark
3-
4-
Benchmark run from 2021-03-24 02:05:58.706995Z UTC
3+
Benchmark run from 2024-05-04 10:49:35.563150Z UTC
54

65
## System
76

@@ -10,22 +9,22 @@ Benchmark suite executing on the following system:
109
<table style="width: 1%">
1110
<tr>
1211
<th style="width: 1%; white-space: nowrap">Operating System</th>
13-
<td>Linux</td>
12+
<td>macOS</td>
1413
</tr><tr>
1514
<th style="white-space: nowrap">CPU Information</th>
16-
<td style="white-space: nowrap">AMD Ryzen 7 PRO 4750U with Radeon Graphics</td>
15+
<td style="white-space: nowrap">Apple M3 Max</td>
1716
</tr><tr>
1817
<th style="white-space: nowrap">Number of Available Cores</th>
1918
<td style="white-space: nowrap">16</td>
2019
</tr><tr>
2120
<th style="white-space: nowrap">Available Memory</th>
22-
<td style="white-space: nowrap">14.92 GB</td>
21+
<td style="white-space: nowrap">128 GB</td>
2322
</tr><tr>
2423
<th style="white-space: nowrap">Elixir Version</th>
25-
<td style="white-space: nowrap">1.11.3</td>
24+
<td style="white-space: nowrap">1.16.2</td>
2625
</tr><tr>
2726
<th style="white-space: nowrap">Erlang Version</th>
28-
<td style="white-space: nowrap">23.2.6</td>
27+
<td style="white-space: nowrap">26.2.4</td>
2928
</tr>
3029
</table>
3130

@@ -50,7 +49,6 @@ Benchmark suite executing with the following configuration:
5049

5150

5251

53-
5452
__Input: Changeset__
5553

5654
Run Time
@@ -67,35 +65,35 @@ Run Time
6765

6866
<tr>
6967
<td style="white-space: nowrap">SQLite3 Insert</td>
70-
<td style="white-space: nowrap; text-align: right">7218.07</td>
71-
<td style="white-space: nowrap; text-align: right">0.139 ms</td>
72-
<td style="white-space: nowrap; text-align: right">±43.60%</td>
73-
<td style="white-space: nowrap; text-align: right">0.123 ms</td>
74-
<td style="white-space: nowrap; text-align: right">0.37 ms</td>
68+
<td style="white-space: nowrap; text-align: right">26.72 K</td>
69+
<td style="white-space: nowrap; text-align: right">37.42 &micro;s</td>
70+
<td style="white-space: nowrap; text-align: right">&plusmn;89.66%</td>
71+
<td style="white-space: nowrap; text-align: right">32.88 &micro;s</td>
72+
<td style="white-space: nowrap; text-align: right">74.21 &micro;s</td>
7573
</tr>
7674

7775
<tr>
7876
<td style="white-space: nowrap">Pg Insert</td>
79-
<td style="white-space: nowrap; text-align: right">421.57</td>
80-
<td style="white-space: nowrap; text-align: right">2.37 ms</td>
81-
<td style="white-space: nowrap; text-align: right">±12.13%</td>
82-
<td style="white-space: nowrap; text-align: right">2.37 ms</td>
83-
<td style="white-space: nowrap; text-align: right">2.90 ms</td>
77+
<td style="white-space: nowrap; text-align: right">9.65 K</td>
78+
<td style="white-space: nowrap; text-align: right">103.63 &micro;s</td>
79+
<td style="white-space: nowrap; text-align: right">&plusmn;68.38%</td>
80+
<td style="white-space: nowrap; text-align: right">102.75 &micro;s</td>
81+
<td style="white-space: nowrap; text-align: right">177.39 &micro;s</td>
8482
</tr>
8583

8684
<tr>
8785
<td style="white-space: nowrap">MyXQL Insert</td>
88-
<td style="white-space: nowrap; text-align: right">284.25</td>
89-
<td style="white-space: nowrap; text-align: right">3.52 ms</td>
90-
<td style="white-space: nowrap; text-align: right">±13.34%</td>
91-
<td style="white-space: nowrap; text-align: right">3.53 ms</td>
92-
<td style="white-space: nowrap; text-align: right">5.05 ms</td>
86+
<td style="white-space: nowrap; text-align: right">5.49 K</td>
87+
<td style="white-space: nowrap; text-align: right">182.25 &micro;s</td>
88+
<td style="white-space: nowrap; text-align: right">&plusmn;49.23%</td>
89+
<td style="white-space: nowrap; text-align: right">182.33 &micro;s</td>
90+
<td style="white-space: nowrap; text-align: right">233.08 &micro;s</td>
9391
</tr>
9492

9593
</table>
9694

9795

98-
Comparison
96+
Run Time Comparison
9997

10098
<table style="width: 1%">
10199
<tr>
@@ -104,28 +102,26 @@ Comparison
104102
<th style="text-align: right">Slower</th>
105103
<tr>
106104
<td style="white-space: nowrap">SQLite3 Insert</td>
107-
<td style="white-space: nowrap;text-align: right">7218.07</td>
105+
<td style="white-space: nowrap;text-align: right">26.72 K</td>
108106
<td>&nbsp;</td>
109107
</tr>
110108

111109
<tr>
112110
<td style="white-space: nowrap">Pg Insert</td>
113-
<td style="white-space: nowrap; text-align: right">421.57</td>
114-
<td style="white-space: nowrap; text-align: right">17.12x</td>
111+
<td style="white-space: nowrap; text-align: right">9.65 K</td>
112+
<td style="white-space: nowrap; text-align: right">2.77x</td>
115113
</tr>
116114

117115
<tr>
118116
<td style="white-space: nowrap">MyXQL Insert</td>
119-
<td style="white-space: nowrap; text-align: right">284.25</td>
120-
<td style="white-space: nowrap; text-align: right">25.39x</td>
117+
<td style="white-space: nowrap; text-align: right">5.49 K</td>
118+
<td style="white-space: nowrap; text-align: right">4.87x</td>
121119
</tr>
122120

123121
</table>
124122

125123

126124

127-
<hr/>
128-
129125

130126
__Input: Struct__
131127

@@ -143,35 +139,35 @@ Run Time
143139

144140
<tr>
145141
<td style="white-space: nowrap">SQLite3 Insert</td>
146-
<td style="white-space: nowrap; text-align: right">7765.76</td>
147-
<td style="white-space: nowrap; text-align: right">0.129 ms</td>
148-
<td style="white-space: nowrap; text-align: right">±32.88%</td>
149-
<td style="white-space: nowrap; text-align: right">0.122 ms</td>
150-
<td style="white-space: nowrap; text-align: right">0.28 ms</td>
142+
<td style="white-space: nowrap; text-align: right">26.71 K</td>
143+
<td style="white-space: nowrap; text-align: right">37.44 &micro;s</td>
144+
<td style="white-space: nowrap; text-align: right">&plusmn;87.15%</td>
145+
<td style="white-space: nowrap; text-align: right">32.92 &micro;s</td>
146+
<td style="white-space: nowrap; text-align: right">70.50 &micro;s</td>
151147
</tr>
152148

153149
<tr>
154150
<td style="white-space: nowrap">Pg Insert</td>
155-
<td style="white-space: nowrap; text-align: right">422.86</td>
156-
<td style="white-space: nowrap; text-align: right">2.36 ms</td>
157-
<td style="white-space: nowrap; text-align: right">±10.49%</td>
158-
<td style="white-space: nowrap; text-align: right">2.36 ms</td>
159-
<td style="white-space: nowrap; text-align: right">3.02 ms</td>
151+
<td style="white-space: nowrap; text-align: right">9.34 K</td>
152+
<td style="white-space: nowrap; text-align: right">107.08 &micro;s</td>
153+
<td style="white-space: nowrap; text-align: right">&plusmn;13.80%</td>
154+
<td style="white-space: nowrap; text-align: right">106.87 &micro;s</td>
155+
<td style="white-space: nowrap; text-align: right">132.46 &micro;s</td>
160156
</tr>
161157

162158
<tr>
163159
<td style="white-space: nowrap">MyXQL Insert</td>
164-
<td style="white-space: nowrap; text-align: right">274.00</td>
165-
<td style="white-space: nowrap; text-align: right">3.65 ms</td>
166-
<td style="white-space: nowrap; text-align: right">±38.43%</td>
167-
<td style="white-space: nowrap; text-align: right">3.59 ms</td>
168-
<td style="white-space: nowrap; text-align: right">4.75 ms</td>
160+
<td style="white-space: nowrap; text-align: right">5.67 K</td>
161+
<td style="white-space: nowrap; text-align: right">176.45 &micro;s</td>
162+
<td style="white-space: nowrap; text-align: right">&plusmn;70.69%</td>
163+
<td style="white-space: nowrap; text-align: right">176.79 &micro;s</td>
164+
<td style="white-space: nowrap; text-align: right">234.70 &micro;s</td>
169165
</tr>
170166

171167
</table>
172168

173169

174-
Comparison
170+
Run Time Comparison
175171

176172
<table style="width: 1%">
177173
<tr>
@@ -180,25 +176,20 @@ Comparison
180176
<th style="text-align: right">Slower</th>
181177
<tr>
182178
<td style="white-space: nowrap">SQLite3 Insert</td>
183-
<td style="white-space: nowrap;text-align: right">7765.76</td>
179+
<td style="white-space: nowrap;text-align: right">26.71 K</td>
184180
<td>&nbsp;</td>
185181
</tr>
186182

187183
<tr>
188184
<td style="white-space: nowrap">Pg Insert</td>
189-
<td style="white-space: nowrap; text-align: right">422.86</td>
190-
<td style="white-space: nowrap; text-align: right">18.37x</td>
185+
<td style="white-space: nowrap; text-align: right">9.34 K</td>
186+
<td style="white-space: nowrap; text-align: right">2.86x</td>
191187
</tr>
192188

193189
<tr>
194190
<td style="white-space: nowrap">MyXQL Insert</td>
195-
<td style="white-space: nowrap; text-align: right">274.00</td>
196-
<td style="white-space: nowrap; text-align: right">28.34x</td>
191+
<td style="white-space: nowrap; text-align: right">5.67 K</td>
192+
<td style="white-space: nowrap; text-align: right">4.71x</td>
197193
</tr>
198194

199-
</table>
200-
201-
202-
203-
<hr/>
204-
195+
</table>

0 commit comments

Comments
 (0)