sql/bulksst: coordinate distributed SST metadata merge with CombineFileInfo #157004
Conversation
|
This change is |
spilchen
force-pushed
the
gh-156662/251105/1157/sst-combine-and-sample/pr-ready
branch
from
f3ed057 to
51fed3f
Compare
spilchen
requested review from
jeffswenson,
kev-cao and
mw5h
and removed request for
a team
mw5h
left a comment
mw5h
left a comment
There was a problem hiding this comment.
@mw5h reviewed 1 of 4 files at r1, all commit messages.
Reviewable status:complete! 0 of 0 LGTMs obtained (waiting on @jeffswenson and @kev-cao)
pkg/sql/bulksst/combine_file_info.go line 17 at r1 (raw file):
) // CombineFileInfo combines SST file metadata and determines merge task spans based on key samples.
Perhaps we should describe what we're producing here and why? There's not really an in-code description of what the samples are, so it's not necessarily clear why we want to use them as break points for our merge spans.
…leInfo Implements CombineFileInfo(), a coordinator utility that aggregates SST metadata from distributed workers and determines merge task spans based on sampled keys. The function combines SST file metadata from multiple workers and uses their row samples to split schema spans into merge task spans. This will be used by the new distributed merge pipeline. Resolves: cockroachdb#156662 Epic: CRDB-48845 Release note: none Co-authored by: @jeffswenson
spilchen
force-pushed
the
gh-156662/251105/1157/sst-combine-and-sample/pr-ready
branch
from
51fed3f to
1fc8f79
Compare
spilchen
left a comment
spilchen
left a comment
There was a problem hiding this comment.
Reviewable status:
pkg/sql/bulksst/combine_file_info.go line 17 at r1 (raw file):
Previously, mw5h (Matt White) wrote…
Perhaps we should describe what we're producing here and why? There's not really an in-code description of what the samples are, so it's not necessarily clear why we want to use them as break points for our merge spans.
Good idea. I beefed up the function comment.
|
@mw5h ready for another look |
mw5h
left a comment
mw5h
left a comment
There was a problem hiding this comment.
LGTM!
@mw5h reviewed 2 of 4 files at r1, 1 of 1 files at r2, all commit messages.
Reviewable status:
|
TFTRs! bors r+ |
4 participants
Implements CombineFileInfo(), a coordinator utility that aggregates SST metadata from distributed workers and determines merge task spans based on sampled keys.
The function combines SST file metadata from multiple workers and uses their row samples to split schema spans into merge task spans. This will be used by the new distributed merge pipeline.
Resolves: #156662
Epic: CRDB-48845
Release note: none
Co-authored by: @jeffswenson