Add support for data manipulation operations in Blob Storage adapter #3124

KodaiD · 2025-11-06T10:14:34Z

Description

This PR adds DML support for the Blob adapter, enabling data manipulation operations on Azure Blob Storage.

Support for S3 and GCS will be addressed in subsequent PRs.

Related issues and/or PRs

Add support for administrative operations in Blob adapter #3104

Changes made

Added DistributedStorage implementation for Object Storage.
Added unit tests.
Added integration tests.

Checklist

I have commented my code, particularly in hard-to-understand areas.
I have updated the documentation to reflect the changes.
I have considered whether similar issues could occur in other products, components, or modules if this PR is for bug fixes.
Any remaining open issues linked to this PR are documented and up-to-date (Jira, GitHub, etc.).
Tests (unit, integration, etc.) have been added for the changes.
My changes generate no new warnings.
Any dependent changes in other PRs have been merged and published.

Additional notes (optional)

N/A

Release notes

Added support for data manipulation operations over Azure Blob Storage.

KodaiD · 2025-11-06T12:53:12Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces data manipulation language (DML) support for the Blob Storage adapter, which is a significant and well-implemented feature. The changes include the core logic for handling mutations, numerous integration tests to ensure correctness, and necessary updates to error codes and utility classes. My review focuses on improving code clarity, robustness, and performance in the new classes. I've identified opportunities to remove dead code, enhance immutability, improve test coverage for conditional operations, and optimize a hashCode implementation. Overall, this is a solid contribution that greatly enhances the capabilities of the object storage adapter.

core/src/main/java/com/scalar/db/storage/objectstorage/ObjectStorageRecord.java

...ava/com/scalar/db/storage/objectstorage/ObjectStorageConditionalMutationIntegrationTest.java

core/src/main/java/com/scalar/db/common/CoreError.java

core/src/main/java/com/scalar/db/storage/objectstorage/ObjectStorageMutation.java

core/src/main/java/com/scalar/db/storage/objectstorage/PartitionIdentifier.java

gemini-code-assist

Code Review

This pull request adds significant data manipulation capabilities to the Blob Storage adapter, a major step forward. The overall implementation is solid, with good use of handlers for different statement types and a clear read-modify-write pattern for mutations. I've identified a few areas for improvement, including a bug in data serialization, a suboptimal hash implementation, and opportunities to enhance test coverage and code readability. My detailed comments are below.

core/src/main/java/com/scalar/db/storage/objectstorage/MapVisitor.java

...ava/com/scalar/db/storage/objectstorage/ObjectStorageConditionalMutationIntegrationTest.java

core/src/main/java/com/scalar/db/common/CoreError.java

core/src/main/java/com/scalar/db/storage/objectstorage/ObjectStorage.java

core/src/main/java/com/scalar/db/storage/objectstorage/PartitionIdentifier.java

gemini-code-assist

Code Review

This pull request introduces Data Manipulation Language (DML) support for the Azure Blob Storage adapter, which is a significant enhancement. The changes include new classes for handling mutations and selections, as well as comprehensive integration and unit tests. The implementation correctly identifies and enforces several limitations inherent to object storage, such as the lack of index support, restricted conditional operations for BLOB types, and limitations on cross-partition scan ordering. The addition of specific error codes for these scenarios is also a positive step for clarity and debugging. Overall, the changes are well-structured and provide a solid foundation for DML operations in Object Storage.

core/src/main/java/com/scalar/db/storage/objectstorage/SelectStatementHandler.java

core/src/main/java/com/scalar/db/storage/objectstorage/MapVisitor.java

core/src/main/java/com/scalar/db/storage/objectstorage/ObjectStorageRecord.java

core/src/main/java/com/scalar/db/storage/objectstorage/ResultInterpreter.java

core/src/integration-test/java/com/scalar/db/storage/objectstorage/ObjectStorageTestUtils.java

Copilot

Copilot reviewed 56 out of 56 changed files in this pull request and generated no comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

...ation-test/java/com/scalar/db/storage/objectstorage/ObjectStorageWrapperIntegrationTest.java

core/src/main/java/com/scalar/db/storage/objectstorage/ClusteringKeyComparator.java

core/src/main/java/com/scalar/db/storage/objectstorage/MutateStatementHandler.java

core/src/main/java/com/scalar/db/storage/objectstorage/StatementHandler.java

core/src/main/java/com/scalar/db/storage/objectstorage/SelectStatementHandler.java

feeblefakie

I took a glance, and the code overall looks OK, but I felt that the class design can be improved.
I left one suggestion. so PTAL.

core/src/main/java/com/scalar/db/storage/objectstorage/MutateStatementHandler.java

core/src/main/java/com/scalar/db/storage/objectstorage/SelectStatementHandler.java

core/src/main/java/com/scalar/db/storage/objectstorage/MutateStatementHandler.java

komamitsu

LGTM! 👍

josh-wong

I've added some minor suggestions. PTAL!

...storage/objectstorage/ConsensusCommitCrossPartitionScanIntegrationTestWithObjectStorage.java

...java/com/scalar/db/storage/objectstorage/ObjectStorageCrossPartitionScanIntegrationTest.java

...objectstorage/TwoPhaseConsensusCommitCrossPartitionScanIntegrationTestWithObjectStorage.java

josh-wong

LGTM! Thank you!🙇🏻‍♂️

brfrn169

Overall LGTM. Left several comments. PTAL!

Just in case, I’ll take another look later.

core/src/main/java/com/scalar/db/storage/objectstorage/SelectStatementHandler.java

core/src/main/java/com/scalar/db/storage/objectstorage/StatementHandler.java

core/src/main/java/com/scalar/db/storage/objectstorage/ObjectStorageMutation.java

core/src/main/java/com/scalar/db/storage/objectstorage/MutateStatementHandler.java

brfrn169

Left several comments. PTAL!

core/src/main/java/com/scalar/db/storage/objectstorage/StatementHandler.java

core/src/main/java/com/scalar/db/storage/objectstorage/StreamingRecordIterator.java

feeblefakie

LGTM! Thank you!

brfrn169

LGTM! Thank you!

feeblefakie · 2025-11-13T05:02:57Z

@Torch3333 Can you check?

Torch3333

LGTM, thank you!

…3124)

Add DML support for Blob Storage

1231cb7

KodaiD self-assigned this Nov 6, 2025

KodaiD added the enhancement New feature or request label Nov 6, 2025

gemini-code-assist bot reviewed Nov 6, 2025

View reviewed changes

Apply suggestions

4811884

KodaiD marked this pull request as ready for review November 7, 2025 01:26

Copilot AI review requested due to automatic review settings November 7, 2025 01:26

Copilot AI reviewed Nov 7, 2025

View reviewed changes

KodaiD requested review from a team, Torch3333, brfrn169, feeblefakie and komamitsu and removed request for a team November 7, 2025 01:44

KodaiD commented Nov 7, 2025

View reviewed changes

...ation-test/java/com/scalar/db/storage/objectstorage/ObjectStorageWrapperIntegrationTest.java Show resolved Hide resolved

Fix to remove BLOB type limitations

0a73daf

KodaiD force-pushed the add-dml-support-for-blob-storage branch from 70f2d31 to 0a73daf Compare November 7, 2025 02:34

Update the OperationChecker unit test to reflect the code change

dcc742e

komamitsu reviewed Nov 7, 2025

View reviewed changes

core/src/main/java/com/scalar/db/storage/objectstorage/ClusteringKeyComparator.java Outdated Show resolved Hide resolved

core/src/main/java/com/scalar/db/storage/objectstorage/MutateStatementHandler.java Show resolved Hide resolved

komamitsu reviewed Nov 7, 2025

View reviewed changes

KodaiD requested a review from komamitsu November 7, 2025 07:25

Fix based on review

533f3cd

feeblefakie reviewed Nov 10, 2025

View reviewed changes

core/src/main/java/com/scalar/db/storage/objectstorage/MutateStatementHandler.java Outdated Show resolved Hide resolved

Apply suggestions

0de9149

KodaiD force-pushed the add-dml-support-for-blob-storage branch from 63647e2 to 0de9149 Compare November 10, 2025 23:10

KodaiD added 2 commits November 11, 2025 08:11

Resolve conflicts

f9c1da5

Add unit tests for ObjectStoragePartition

e0657fd

KodaiD force-pushed the add-dml-support-for-blob-storage branch from 8cef4ad to 493419b Compare November 11, 2025 01:44

feeblefakie reviewed Nov 11, 2025

View reviewed changes

brfrn169 requested review from josh-wong and removed request for brfrn169 November 11, 2025 03:45

KodaiD added 2 commits November 11, 2025 15:03

Apply suggestions

c8e1cf0

Fix integration tests

05c831b

KodaiD requested review from brfrn169 and feeblefakie November 11, 2025 06:09

komamitsu approved these changes Nov 11, 2025

View reviewed changes

josh-wong requested changes Nov 11, 2025

View reviewed changes

Apply suggestions

7228c1a

KodaiD requested a review from josh-wong November 11, 2025 07:20

josh-wong approved these changes Nov 11, 2025

View reviewed changes

brfrn169 reviewed Nov 11, 2025

View reviewed changes

KodaiD added 2 commits November 11, 2025 21:10

Apply suggestions

669501a

Fix to suppress warnings

1ca754a

KodaiD requested a review from brfrn169 November 11, 2025 23:17

KodaiD mentioned this pull request Nov 12, 2025

Add support for S3 #3141

Open

7 tasks

brfrn169 reviewed Nov 12, 2025

View reviewed changes

Apply suggestions

2297d82

KodaiD requested a review from brfrn169 November 12, 2025 05:41

feeblefakie approved these changes Nov 12, 2025

View reviewed changes

brfrn169 approved these changes Nov 12, 2025

View reviewed changes

Torch3333 approved these changes Nov 13, 2025

View reviewed changes

Merge branch 'master' into add-dml-support-for-blob-storage

781c9c7

KodaiD merged commit ba20b01 into master Nov 13, 2025
203 of 208 checks passed

KodaiD deleted the add-dml-support-for-blob-storage branch November 13, 2025 08:18

feeblefakie pushed a commit that referenced this pull request Nov 13, 2025

Add support for data manipulation operations in Blob Storage adapter (#…

e0fa714

…3124)

feeblefakie mentioned this pull request Nov 13, 2025

Backport to branch(3) : Add support for data manipulation operations in Blob Storage adapter #3158

Merged

Add support for data manipulation operations in Blob Storage adapter #3124

Add support for data manipulation operations in Blob Storage adapter #3124

Conversation

KodaiD commented Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related issues and/or PRs

Changes made

Checklist

Additional notes (optional)

Release notes

Uh oh!

KodaiD commented Nov 6, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

feeblefakie left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

komamitsu left a comment

Choose a reason for hiding this comment

Uh oh!

josh-wong left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

josh-wong left a comment

Choose a reason for hiding this comment

Uh oh!

brfrn169 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

KodaiD commented Nov 6, 2025 •

edited

Loading

brfrn169 left a comment •

edited

Loading