Switching to a fixed CFS threshold #15295
Conversation
| } | ||
|
|
||
| // Apply appropriate threshold based on merge policy type | ||
| if (mergePolicy instanceof LogDocMergePolicy) { |
There was a problem hiding this comment.
It would be great if we can avoid customizing it for specific policies, otherwise it might be tricky to maintain in the future, if e.g. there is another policy that is based on doc size not bytes.
Maybe we can add a enum and a method to MergePolicy which returns its unit (bytes/docs) , and use it here to decide which threshold to use?
Or do we want to always choose compound format based on size in bytes even for LogDocMergePolicy? In this case we might be able to use merge.getMergeInfo().sizeInBytes() when we call this method and avoid relying on MergePolicy#size all together?
There was a problem hiding this comment.
+1 to the idea of Enums, I will wait if anyone else has other suggestions here but having an enum makes most sense to me.
|
This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution! |
stefanvodita
left a comment
stefanvodita
left a comment
There was a problem hiding this comment.
I had looked at this PR before, but I see it's gone stale. Hoping to revive it.
Surprised how many changes we needed for this!
lucene/core/src/java/org/apache/lucene/index/LiveIndexWriterConfig.java
Outdated
Show resolved
Hide resolved
lucene/core/src/test/org/apache/lucene/index/TestAddIndexes.java
Outdated
Show resolved
Hide resolved
lucene/test-framework/src/java/org/apache/lucene/tests/util/LuceneTestCase.java
Show resolved
Hide resolved
stefanvodita
left a comment
stefanvodita
left a comment
There was a problem hiding this comment.
Thank you for addressing my comments! There are two questions left for me from the comments:
- Whether the thresholds we're starting with are good enough for a first iteration.
- Whether we can avoid branching logic based on which merge policy is in use.
| * MergePolicy#setNoCFSRatio(double)} and {@link MergePolicy#setMaxCFSSegmentSizeMB(double)}. This | ||
| * setting only applies to newly created segments.</b> | ||
| * <p><b>Note: To control compound file usage during segment merges. More here: | ||
| * lucene/core/src/java/org/apache/lucene/codecs/CompoundFormat.java</b> |
There was a problem hiding this comment.
Can we use the @link syntax?
3 participants
Problem
Lucene uses CFS ratio (which is by default 10%) to determine whether to use Compound Files.
Solution
In this PR we are doing the following:
Switching from a CFS ratio to a fixed threshold which is:
Moved CFS configuration to CompoundFormat.java from Merge policies.
Fixes #14959