Commit cd04743
committed
[KYUUBI #7175] Enhance the performance for metadata cleanup by leveraging
### Why are the changes needed?
First, `create_time < end_time` is always true.
So, when cleaning the mtadata, minEndTime = System.currentTimeMillis - maxAge.
end_time < minEndTime.
And due to `create_time < end_time`, so `create_time < minEndTime` is also true.
And with `create_time < minEndTime`, its performance is better because it leverages `create_time` index.
https://github.com/apache/kyuubi/blob/f7e10e65d3aca6fa82171bb3d75b7622c74807b7/kyuubi-server/src/main/resources/sql/sqlite/metadata-store-schema-1.11.0.sqlite.sql#L37
```
mysql> select count(1) from metadata where state in ('FINISHED', 'TIMEOUT', 'CANCELED', 'CLOSED', 'ERROR') and end_time < 1753118619315;
+----------+
| count(1) |
+----------+
| 27741 |
+----------+
1 row in set (2.40 sec)
mysql> select count(1) from metadata where state in ('FINISHED', 'TIMEOUT', 'CANCELED', 'CLOSED', 'ERROR') and end_time < 1753118619315 and create_time < 1753118619315;
+----------+
| count(1) |
+----------+
| 27741 |
+----------+
1 row in set (0.18 sec)
mysql> select count(1) from metadata where end_time>0 and end_time < 1753118619315 and create_time < 1753118619315;
+----------+
| count(1) |
+----------+
| 27741 |
+----------+
1 row in set (0.13 sec)
```
### How was this patch tested?
GA and manually testing.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #7175 from turboFei/delete_fast.
Closes #7175
980a391 [Wang, Fei] delete fast
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>create_time index1 parent d9e0d36 commit cd04743
File tree
1 file changed
+3
-4
lines changed- kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc
1 file changed
+3
-4
lines changedLines changed: 3 additions & 4 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
84 | 84 | | |
85 | 85 | | |
86 | 86 | | |
87 | | - | |
88 | | - | |
89 | 87 | | |
90 | 88 | | |
91 | 89 | | |
| |||
413 | 411 | | |
414 | 412 | | |
415 | 413 | | |
416 | | - | |
| 414 | + | |
| 415 | + | |
417 | 416 | | |
418 | | - | |
| 417 | + | |
419 | 418 | | |
420 | 419 | | |
421 | 420 | | |
| |||
0 commit comments