Skip to content

Commit cd04743

Browse files
committed
[KYUUBI #7175] Enhance the performance for metadata cleanup by leveraging create_time index
### Why are the changes needed? First, `create_time < end_time` is always true. So, when cleaning the mtadata, minEndTime = System.currentTimeMillis - maxAge. end_time < minEndTime. And due to `create_time < end_time`, so `create_time < minEndTime` is also true. And with `create_time < minEndTime`, its performance is better because it leverages `create_time` index. https://github.com/apache/kyuubi/blob/f7e10e65d3aca6fa82171bb3d75b7622c74807b7/kyuubi-server/src/main/resources/sql/sqlite/metadata-store-schema-1.11.0.sqlite.sql#L37 ``` mysql> select count(1) from metadata where state in ('FINISHED', 'TIMEOUT', 'CANCELED', 'CLOSED', 'ERROR') and end_time < 1753118619315; +----------+ | count(1) | +----------+ | 27741 | +----------+ 1 row in set (2.40 sec) mysql> select count(1) from metadata where state in ('FINISHED', 'TIMEOUT', 'CANCELED', 'CLOSED', 'ERROR') and end_time < 1753118619315 and create_time < 1753118619315; +----------+ | count(1) | +----------+ | 27741 | +----------+ 1 row in set (0.18 sec) mysql> select count(1) from metadata where end_time>0 and end_time < 1753118619315 and create_time < 1753118619315; +----------+ | count(1) | +----------+ | 27741 | +----------+ 1 row in set (0.13 sec) ``` ### How was this patch tested? GA and manually testing. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #7175 from turboFei/delete_fast. Closes #7175 980a391 [Wang, Fei] delete fast Authored-by: Wang, Fei <fwang12@ebay.com> Signed-off-by: Wang, Fei <fwang12@ebay.com>
1 parent d9e0d36 commit cd04743

File tree

1 file changed

+3
-4
lines changed

1 file changed

+3
-4
lines changed

kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStore.scala

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -84,8 +84,6 @@ class JDBCMetadataStore(conf: KyuubiConf) extends MetadataStore with Logging {
8484
new HikariDataSource(hikariConfig)
8585
private val mapper = new ObjectMapper().registerModule(DefaultScalaModule)
8686

87-
private val terminalStates = OperationState.terminalStates.map(x => s"'$x'").mkString(", ")
88-
8987
if (conf.get(METADATA_STORE_JDBC_DATABASE_SCHEMA_INIT)) {
9088
initSchema()
9189
}
@@ -413,9 +411,10 @@ class JDBCMetadataStore(conf: KyuubiConf) extends MetadataStore with Logging {
413411

414412
override def cleanupMetadataByAge(maxAge: Long): Unit = {
415413
val minEndTime = System.currentTimeMillis() - maxAge
416-
val query = s"DELETE FROM $METADATA_TABLE WHERE state IN ($terminalStates) AND end_time < ?"
414+
val query =
415+
s"DELETE FROM $METADATA_TABLE WHERE end_time > 0 AND end_time < ? AND create_time < ?"
417416
JdbcUtils.withConnection { connection =>
418-
withUpdateCount(connection, query, minEndTime) { count =>
417+
withUpdateCount(connection, query, minEndTime, minEndTime) { count =>
419418
info(s"Cleaned up $count records older than $maxAge ms from $METADATA_TABLE.")
420419
}
421420
}

0 commit comments

Comments
 (0)