This repository was archived by the owner on May 17, 2024. It is now read-only.
Commit b3d4223
Make dbt data diffs concurrent (#776)
* v0 of concurrency
* concurrent logging
* remove todo
* remove todo
* better var name
* add node name to logger
* format string logs
* add optional logger param
* avoid extra threads
* use thread pools
* not multithreaded at the connection level anymore
* show errors as they happen
* show full stacktrace on error
* rearrange trace
* more logs for debugging
* update for threads mocking
* clear log params
* remove extra space
* remove long traceback
* Ensure log_message is optional
Co-authored-by: Dan Lawin <daniel@datafold.com>
* map threaded result to proper model id
* explicit type and optional
* rm submodules again
---------
Co-authored-by: Sung Won Chung <sung@datafold.com>
Co-authored-by: Dan Lawin <daniel@datafold.com>1 parent 29b48b0 commit b3d4223
File tree
6 files changed
+110
-64
lines changed- data_diff
- databases
- tests
6 files changed
+110
-64
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
931 | 931 | | |
932 | 932 | | |
933 | 933 | | |
934 | | - | |
| 934 | + | |
935 | 935 | | |
936 | 936 | | |
937 | 937 | | |
| |||
956 | 956 | | |
957 | 957 | | |
958 | 958 | | |
959 | | - | |
| 959 | + | |
| 960 | + | |
| 961 | + | |
| 962 | + | |
960 | 963 | | |
961 | 964 | | |
962 | 965 | | |
| |||
1022 | 1025 | | |
1023 | 1026 | | |
1024 | 1027 | | |
1025 | | - | |
| 1028 | + | |
1026 | 1029 | | |
1027 | 1030 | | |
1028 | 1031 | | |
| |||
1044 | 1047 | | |
1045 | 1048 | | |
1046 | 1049 | | |
1047 | | - | |
| 1050 | + | |
1048 | 1051 | | |
1049 | 1052 | | |
1050 | 1053 | | |
| |||
1086 | 1089 | | |
1087 | 1090 | | |
1088 | 1091 | | |
1089 | | - | |
| 1092 | + | |
| 1093 | + | |
| 1094 | + | |
1090 | 1095 | | |
1091 | 1096 | | |
1092 | 1097 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| 11 | + | |
11 | 12 | | |
12 | 13 | | |
13 | 14 | | |
| |||
80 | 81 | | |
81 | 82 | | |
82 | 83 | | |
83 | | - | |
84 | 84 | | |
85 | 85 | | |
86 | 86 | | |
| |||
112 | 112 | | |
113 | 113 | | |
114 | 114 | | |
115 | | - | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
116 | 120 | | |
117 | 121 | | |
118 | 122 | | |
| |||
140 | 144 | | |
141 | 145 | | |
142 | 146 | | |
143 | | - | |
| 147 | + | |
144 | 148 | | |
145 | 149 | | |
146 | | - | |
147 | 150 | | |
148 | | - | |
| 151 | + | |
| 152 | + | |
149 | 153 | | |
150 | 154 | | |
151 | 155 | | |
| |||
165 | 169 | | |
166 | 170 | | |
167 | 171 | | |
168 | | - | |
169 | | - | |
170 | | - | |
171 | | - | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
172 | 178 | | |
173 | 179 | | |
174 | 180 | | |
| |||
265 | 271 | | |
266 | 272 | | |
267 | 273 | | |
268 | | - | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
269 | 279 | | |
270 | 280 | | |
271 | 281 | | |
272 | 282 | | |
273 | | - | |
274 | | - | |
275 | | - | |
276 | | - | |
| 283 | + | |
| 284 | + | |
277 | 285 | | |
278 | 286 | | |
279 | 287 | | |
| |||
373 | 381 | | |
374 | 382 | | |
375 | 383 | | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
376 | 387 | | |
377 | 388 | | |
378 | 389 | | |
| |||
406 | 417 | | |
407 | 418 | | |
408 | 419 | | |
409 | | - | |
| 420 | + | |
410 | 421 | | |
411 | 422 | | |
412 | 423 | | |
| |||
476 | 487 | | |
477 | 488 | | |
478 | 489 | | |
479 | | - | |
| 490 | + | |
480 | 491 | | |
481 | 492 | | |
482 | 493 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
446 | 446 | | |
447 | 447 | | |
448 | 448 | | |
449 | | - | |
| 449 | + | |
450 | 450 | | |
451 | 451 | | |
452 | 452 | | |
453 | 453 | | |
454 | | - | |
| 454 | + | |
455 | 455 | | |
456 | 456 | | |
457 | 457 | | |
458 | 458 | | |
459 | | - | |
| 459 | + | |
460 | 460 | | |
461 | 461 | | |
462 | 462 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
162 | 162 | | |
163 | 163 | | |
164 | 164 | | |
165 | | - | |
| 165 | + | |
166 | 166 | | |
167 | 167 | | |
168 | 168 | | |
| |||
193 | 193 | | |
194 | 194 | | |
195 | 195 | | |
196 | | - | |
197 | | - | |
| 196 | + | |
| 197 | + | |
198 | 198 | | |
199 | 199 | | |
200 | 200 | | |
| |||
205 | 205 | | |
206 | 206 | | |
207 | 207 | | |
208 | | - | |
209 | | - | |
| 208 | + | |
| 209 | + | |
210 | 210 | | |
211 | 211 | | |
212 | 212 | | |
| |||
227 | 227 | | |
228 | 228 | | |
229 | 229 | | |
230 | | - | |
| 230 | + | |
231 | 231 | | |
232 | 232 | | |
233 | 233 | | |
| |||
240 | 240 | | |
241 | 241 | | |
242 | 242 | | |
243 | | - | |
| 243 | + | |
244 | 244 | | |
245 | 245 | | |
246 | 246 | | |
247 | | - | |
| 247 | + | |
248 | 248 | | |
249 | 249 | | |
250 | 250 | | |
251 | 251 | | |
252 | | - | |
| 252 | + | |
253 | 253 | | |
254 | 254 | | |
255 | 255 | | |
256 | 256 | | |
257 | 257 | | |
258 | 258 | | |
259 | 259 | | |
260 | | - | |
| 260 | + | |
261 | 261 | | |
262 | 262 | | |
263 | 263 | | |
| |||
267 | 267 | | |
268 | 268 | | |
269 | 269 | | |
270 | | - | |
| 270 | + | |
271 | 271 | | |
272 | 272 | | |
273 | 273 | | |
| |||
288 | 288 | | |
289 | 289 | | |
290 | 290 | | |
291 | | - | |
| 291 | + | |
292 | 292 | | |
293 | 293 | | |
294 | 294 | | |
| |||
303 | 303 | | |
304 | 304 | | |
305 | 305 | | |
306 | | - | |
| 306 | + | |
307 | 307 | | |
308 | 308 | | |
309 | 309 | | |
| |||
334 | 334 | | |
335 | 335 | | |
336 | 336 | | |
337 | | - | |
338 | | - | |
339 | | - | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
340 | 353 | | |
341 | 354 | | |
342 | 355 | | |
343 | 356 | | |
344 | 357 | | |
345 | | - | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
346 | 367 | | |
347 | 368 | | |
348 | 369 | | |
349 | 370 | | |
350 | 371 | | |
351 | 372 | | |
352 | | - | |
353 | | - | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
354 | 377 | | |
355 | 378 | | |
356 | 379 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
485 | 485 | | |
486 | 486 | | |
487 | 487 | | |
488 | | - | |
| 488 | + | |
489 | 489 | | |
490 | 490 | | |
491 | 491 | | |
492 | | - | |
493 | | - | |
| 492 | + | |
| 493 | + | |
494 | 494 | | |
495 | 495 | | |
496 | 496 | | |
497 | 497 | | |
498 | 498 | | |
499 | 499 | | |
500 | | - | |
501 | | - | |
502 | | - | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
503 | 503 | | |
504 | | - | |
505 | | - | |
506 | | - | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
507 | 507 | | |
508 | | - | |
509 | | - | |
510 | | - | |
511 | | - | |
512 | | - | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
513 | 513 | | |
514 | 514 | | |
515 | 515 | | |
| |||
0 commit comments