Skip to content

Commit 3ef1bf4

Browse files
kevinlangkevinlang
authored
Update benchmarks, run them, check in results as markdown files (#18)
* update benchmarks, run all but the all one and check in results * format * clean up lockfile
1 parent a3569e2 commit 3ef1bf4

File tree

14 files changed

+1524
-42
lines changed

14 files changed

+1524
-42
lines changed

‎.formatter.exs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
[
33
inputs: [
44
"{mix,.formatter}.exs",
5-
"{lib,test}/**/*.{ex,exs}"
5+
"{lib,test,bench}/**/*.{ex,exs}"
66
],
77
line_length: 88
88
]

‎README.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,3 +39,9 @@ config :my_app,
3939
config :my_app, MyApp.Repo,
4040
database: "path/to/my/database.db",
4141
```
42+
43+
## Benchmarks
44+
45+
We have some benchmarks comparing it against the `MySQL` and `Postgres` adapters.
46+
47+
You can read more about those at [bench/README.md](bench/README.md).

‎bench/README.md‎

Lines changed: 16 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,17 @@
11
# Ecto Benchmarks
22

3+
## Results
4+
5+
| Benchmark | Description |
6+
| --------- | ------------ |
7+
| [load](results/load.md) | tracks performance of loading "raw" data into ecto structures |
8+
| [to_sql](results/to_sql.md) | tracks performance of parsing `Ecto.Query` structures into "raw" SQL query strings |
9+
| [insert](results/insert.md) | tracks performance of inserting changesets and structs in the database with `Repo.insert!/1` function |
10+
11+
For reference, these results were run with a Sabrent Rocket Q 1TB NVMe SSD.
12+
13+
## Running the benchmarks
14+
315
Ecto has a benchmark suite to track performance of sensitive operations. Benchmarks
416
are run using the [Benchee](https://github.com/PragTob/benchee) library and
517
need PostgreSQL and MySQL up and running.
@@ -28,9 +40,10 @@ more likely to integration tests.
2840
You can also run a benchmark individually by giving the path to the benchmark
2941
script instead of `bench/bench_helper.exs`.
3042

31-
# Docker
32-
I had Postgres already installed and running locally, but needed to get MySQL up and running. The easiest way to do this is with this command:
43+
### Docker
44+
The easiest way to setup mysql and postgresql for the benchmarks is via Docker. Run the following commands to get an instance of each running.
3345

3446
```
35-
docker run -p 3306:3306 --name mysql_server -e MYSQL_ALLOW_EMPTY_PASSWORD=yes mysql:5.7
47+
docker run -p 3306:3306 -e MYSQL_ALLOW_EMPTY_PASSWORD=yes mysql:5.7
48+
docker run -p 5432:5432 -e POSTGRES_PASSWORD=postgres postgres:13.2
3649
```

‎bench/results/insert.md‎

Lines changed: 204 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,204 @@
1+
2+
# Benchmark
3+
4+
Benchmark run from 2021-03-24 02:05:58.706995Z UTC
5+
6+
## System
7+
8+
Benchmark suite executing on the following system:
9+
10+
<table style="width: 1%">
11+
<tr>
12+
<th style="width: 1%; white-space: nowrap">Operating System</th>
13+
<td>Linux</td>
14+
</tr><tr>
15+
<th style="white-space: nowrap">CPU Information</th>
16+
<td style="white-space: nowrap">AMD Ryzen 7 PRO 4750U with Radeon Graphics</td>
17+
</tr><tr>
18+
<th style="white-space: nowrap">Number of Available Cores</th>
19+
<td style="white-space: nowrap">16</td>
20+
</tr><tr>
21+
<th style="white-space: nowrap">Available Memory</th>
22+
<td style="white-space: nowrap">14.92 GB</td>
23+
</tr><tr>
24+
<th style="white-space: nowrap">Elixir Version</th>
25+
<td style="white-space: nowrap">1.11.3</td>
26+
</tr><tr>
27+
<th style="white-space: nowrap">Erlang Version</th>
28+
<td style="white-space: nowrap">23.2.6</td>
29+
</tr>
30+
</table>
31+
32+
## Configuration
33+
34+
Benchmark suite executing with the following configuration:
35+
36+
<table style="width: 1%">
37+
<tr>
38+
<th style="width: 1%">:time</th>
39+
<td style="white-space: nowrap">5 s</td>
40+
</tr><tr>
41+
<th>:parallel</th>
42+
<td style="white-space: nowrap">1</td>
43+
</tr><tr>
44+
<th>:warmup</th>
45+
<td style="white-space: nowrap">2 s</td>
46+
</tr>
47+
</table>
48+
49+
## Statistics
50+
51+
52+
53+
54+
__Input: Changeset__
55+
56+
Run Time
57+
58+
<table style="width: 1%">
59+
<tr>
60+
<th>Name</th>
61+
<th style="text-align: right">IPS</th>
62+
<th style="text-align: right">Average</th>
63+
<th style="text-align: right">Devitation</th>
64+
<th style="text-align: right">Median</th>
65+
<th style="text-align: right">99th&nbsp;%</th>
66+
</tr>
67+
68+
<tr>
69+
<td style="white-space: nowrap">SQLite3 Insert</td>
70+
<td style="white-space: nowrap; text-align: right">7218.07</td>
71+
<td style="white-space: nowrap; text-align: right">0.139 ms</td>
72+
<td style="white-space: nowrap; text-align: right">±43.60%</td>
73+
<td style="white-space: nowrap; text-align: right">0.123 ms</td>
74+
<td style="white-space: nowrap; text-align: right">0.37 ms</td>
75+
</tr>
76+
77+
<tr>
78+
<td style="white-space: nowrap">Pg Insert</td>
79+
<td style="white-space: nowrap; text-align: right">421.57</td>
80+
<td style="white-space: nowrap; text-align: right">2.37 ms</td>
81+
<td style="white-space: nowrap; text-align: right">±12.13%</td>
82+
<td style="white-space: nowrap; text-align: right">2.37 ms</td>
83+
<td style="white-space: nowrap; text-align: right">2.90 ms</td>
84+
</tr>
85+
86+
<tr>
87+
<td style="white-space: nowrap">MyXQL Insert</td>
88+
<td style="white-space: nowrap; text-align: right">284.25</td>
89+
<td style="white-space: nowrap; text-align: right">3.52 ms</td>
90+
<td style="white-space: nowrap; text-align: right">±13.34%</td>
91+
<td style="white-space: nowrap; text-align: right">3.53 ms</td>
92+
<td style="white-space: nowrap; text-align: right">5.05 ms</td>
93+
</tr>
94+
95+
</table>
96+
97+
98+
Comparison
99+
100+
<table style="width: 1%">
101+
<tr>
102+
<th>Name</th>
103+
<th style="text-align: right">IPS</th>
104+
<th style="text-align: right">Slower</th>
105+
<tr>
106+
<td style="white-space: nowrap">SQLite3 Insert</td>
107+
<td style="white-space: nowrap;text-align: right">7218.07</td>
108+
<td>&nbsp;</td>
109+
</tr>
110+
111+
<tr>
112+
<td style="white-space: nowrap">Pg Insert</td>
113+
<td style="white-space: nowrap; text-align: right">421.57</td>
114+
<td style="white-space: nowrap; text-align: right">17.12x</td>
115+
</tr>
116+
117+
<tr>
118+
<td style="white-space: nowrap">MyXQL Insert</td>
119+
<td style="white-space: nowrap; text-align: right">284.25</td>
120+
<td style="white-space: nowrap; text-align: right">25.39x</td>
121+
</tr>
122+
123+
</table>
124+
125+
126+
127+
<hr/>
128+
129+
130+
__Input: Struct__
131+
132+
Run Time
133+
134+
<table style="width: 1%">
135+
<tr>
136+
<th>Name</th>
137+
<th style="text-align: right">IPS</th>
138+
<th style="text-align: right">Average</th>
139+
<th style="text-align: right">Devitation</th>
140+
<th style="text-align: right">Median</th>
141+
<th style="text-align: right">99th&nbsp;%</th>
142+
</tr>
143+
144+
<tr>
145+
<td style="white-space: nowrap">SQLite3 Insert</td>
146+
<td style="white-space: nowrap; text-align: right">7765.76</td>
147+
<td style="white-space: nowrap; text-align: right">0.129 ms</td>
148+
<td style="white-space: nowrap; text-align: right">±32.88%</td>
149+
<td style="white-space: nowrap; text-align: right">0.122 ms</td>
150+
<td style="white-space: nowrap; text-align: right">0.28 ms</td>
151+
</tr>
152+
153+
<tr>
154+
<td style="white-space: nowrap">Pg Insert</td>
155+
<td style="white-space: nowrap; text-align: right">422.86</td>
156+
<td style="white-space: nowrap; text-align: right">2.36 ms</td>
157+
<td style="white-space: nowrap; text-align: right">±10.49%</td>
158+
<td style="white-space: nowrap; text-align: right">2.36 ms</td>
159+
<td style="white-space: nowrap; text-align: right">3.02 ms</td>
160+
</tr>
161+
162+
<tr>
163+
<td style="white-space: nowrap">MyXQL Insert</td>
164+
<td style="white-space: nowrap; text-align: right">274.00</td>
165+
<td style="white-space: nowrap; text-align: right">3.65 ms</td>
166+
<td style="white-space: nowrap; text-align: right">±38.43%</td>
167+
<td style="white-space: nowrap; text-align: right">3.59 ms</td>
168+
<td style="white-space: nowrap; text-align: right">4.75 ms</td>
169+
</tr>
170+
171+
</table>
172+
173+
174+
Comparison
175+
176+
<table style="width: 1%">
177+
<tr>
178+
<th>Name</th>
179+
<th style="text-align: right">IPS</th>
180+
<th style="text-align: right">Slower</th>
181+
<tr>
182+
<td style="white-space: nowrap">SQLite3 Insert</td>
183+
<td style="white-space: nowrap;text-align: right">7765.76</td>
184+
<td>&nbsp;</td>
185+
</tr>
186+
187+
<tr>
188+
<td style="white-space: nowrap">Pg Insert</td>
189+
<td style="white-space: nowrap; text-align: right">422.86</td>
190+
<td style="white-space: nowrap; text-align: right">18.37x</td>
191+
</tr>
192+
193+
<tr>
194+
<td style="white-space: nowrap">MyXQL Insert</td>
195+
<td style="white-space: nowrap; text-align: right">274.00</td>
196+
<td style="white-space: nowrap; text-align: right">28.34x</td>
197+
</tr>
198+
199+
</table>
200+
201+
202+
203+
<hr/>
204+

0 commit comments

Comments
 (0)