Deprecate `include-metadata` option for data loader export #3159

ypeckstadt · 2025-11-13T08:51:13Z

Description

This PR deprecates the include-metadata argument in the Data Loader export command. Previously, this flag controlled whether before_ metadata fields were emitted in the export output. This behavior is no longer appropriate because metadata inclusion should be determined solely by the scalar.db.consensus_commit.include_metadata.enabled configuration.

Instead of manually toggling metadata output at the CLI level, the Data Loader should simply export the data exactly as returned by the underlying scan operation. This ensures consistent behavior with how ScalarDB is configured and avoids divergent or confusing export logic.

I kept the argument as mentioned in the code comments, but of course, it is also possible to just remove it altogether.

Related issues and/or PRs

This PR is currently targeting the feat/data-loader/import-replace-storage branch, as many PRs are open that this one is built on. I will update this PR to target master branch when the others are reviewed.

Changes made

Deprecated the include-metadata argument
- It has been deprecated and not removed, so that users who do specify it still receive a warning instead of an exception.
Removed all usage of the include metadata flag in the Data Loader core code
Removed all code in the core code that was doing any filtering or populating for this flag.
Added unit tests

Checklist

The following is a best-effort checklist. If any items in this checklist are not applicable to this PR or are dependent on other, unmerged PRs, please still mark the checkboxes after you have read and understood each item.

I have commented my code, particularly in hard-to-understand areas.
I have updated the documentation to reflect the changes.
I have considered whether similar issues could occur in other products, components, or modules if this PR is for bug fixes.
Any remaining open issues linked to this PR are documented and up-to-date (Jira, GitHub, etc.).
Tests (unit, integration, etc.) have been added for the changes.
My changes generate no new warnings.
Any dependent changes in other PRs have been merged and published.

Additional notes (optional)

NA

Release notes

Deprecated include-metadata option for Data Loader export

… config # Conflicts: # data-loader/cli/src/main/java/com/scalar/db/dataloader/cli/command/dataexport/ExportCommand.java # data-loader/core/src/test/java/com/scalar/db/dataloader/core/dataexport/CsvExportManagerTest.java

gemini-code-assist · 2025-11-13T08:51:44Z

Summary of Changes

Hello @ypeckstadt, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request refactors the Data Loader export mechanism by deprecating the --include-metadata command-line option. The change aims to centralize the control of transaction metadata inclusion to a configuration property, thereby standardizing behavior and simplifying the CLI interface. While the option is no longer functional, it remains present to provide a warning to users, preventing immediate breakage of existing scripts and allowing for a graceful migration to the new configuration-driven approach.

Highlights

Deprecation of include-metadata option: The --include-metadata command-line option for the Data Loader export functionality has been deprecated. This option previously controlled the inclusion of before_ fields in the output.
Centralized Metadata Control: Control over whether transaction metadata is included in scan operations is now managed solely by the scalar.db.consensus_commit.include_metadata.enabled configuration property within the ScalarDB properties file.
Warning for Deprecated Option Usage: Users who continue to specify the --include-metadata argument will receive a warning message instead of an exception, ensuring backward compatibility and a smoother transition.
Removal of Internal Logic: All core code responsible for filtering or populating data based on the include-metadata flag has been removed, simplifying the codebase and removing redundant logic.
Unit Test Updates: New unit tests have been added to specifically verify that the deprecated option is correctly parsed and triggers the intended warning, and that its absence is also correctly handled.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request effectively deprecates the --include-metadata command-line option in favor of a centralized configuration property. The changes are well-structured, removing the now-obsolete logic from the core data loader and producer tasks, and replacing it with a clear warning for users who still use the deprecated flag. The accompanying unit tests are thorough and validate the new warning mechanism. My review includes a few suggestions for minor refactoring to improve code clarity and maintainability, as well as recommendations to clean up some redundant tests that resulted from this change.

...-loader/cli/src/main/java/com/scalar/db/dataloader/cli/command/dataexport/ExportCommand.java

.../cli/src/main/java/com/scalar/db/dataloader/cli/command/dataexport/ExportCommandOptions.java

...der/cli/src/test/java/com/scalar/db/dataloader/cli/command/dataexport/ExportCommandTest.java

data-loader/core/src/main/java/com/scalar/db/dataloader/core/dataexport/CsvExportManager.java

…t() multiple times[skip ci]

komamitsu

LGTM! 👍

brfrn169

LGTM! Thank you!

Torch3333

LGTM, thank you!

thongdk8

LGTM! Thank you!

feeblefakie

LGTM! Thank you!

ypeckstadt added 6 commits November 13, 2025 17:28

WPI

c44db24

Log warning if --include-metadata is used

4989d9d

Improve deprecated warning message

c1fc7b9

Modify deprecated warning

efb1a4d

Simplify unit tests

a007126

ypeckstadt added the improvement label Nov 13, 2025

ypeckstadt self-assigned this Nov 13, 2025

ypeckstadt requested a review from thongdk8 November 13, 2025 08:53

gemini-code-assist bot reviewed Nov 13, 2025

View reviewed changes

ypeckstadt added 4 commits November 13, 2025 17:59

Replace dedicated method with inline coding[skip ci]

5969869

Use assertTrue/assertFalse instead of assertEquals[skip ci]

17011d0

mprove readability and avoid calling spec.commandLine().getParseResul…

e19e2cd

…t() multiple times[skip ci]

Fix typo[skip ci]

b28c8b8

ypeckstadt requested review from a team, Torch3333, brfrn169, feeblefakie and komamitsu and removed request for a team November 13, 2025 09:19

komamitsu approved these changes Nov 14, 2025

View reviewed changes

brfrn169 approved these changes Nov 14, 2025

View reviewed changes

Torch3333 approved these changes Nov 14, 2025

View reviewed changes

thongdk8 approved these changes Nov 14, 2025

View reviewed changes

feeblefakie approved these changes Nov 14, 2025

View reviewed changes

feeblefakie merged commit 96ba8f0 into feat/data-loader/import-replace-storage Nov 14, 2025
3 checks passed

feeblefakie deleted the deprecate/include-metadata-export branch November 14, 2025 05:32

brfrn169 removed the improvement label Nov 14, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Deprecate `include-metadata` option for data loader export #3159

Deprecate `include-metadata` option for data loader export #3159

Uh oh!

ypeckstadt commented Nov 13, 2025 •

edited

Loading

Uh oh!

gemini-code-assist bot commented Nov 13, 2025

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

komamitsu left a comment

Uh oh!

brfrn169 left a comment

Uh oh!

Torch3333 left a comment

Uh oh!

thongdk8 left a comment

Uh oh!

feeblefakie left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Deprecate include-metadata option for data loader export #3159

Deprecate include-metadata option for data loader export #3159

Uh oh!

Conversation

ypeckstadt commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related issues and/or PRs

Changes made

Checklist

Additional notes (optional)

Release notes

Uh oh!

gemini-code-assist bot commented Nov 13, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

komamitsu left a comment

Choose a reason for hiding this comment

Uh oh!

brfrn169 left a comment

Choose a reason for hiding this comment

Uh oh!

Torch3333 left a comment

Choose a reason for hiding this comment

Uh oh!

thongdk8 left a comment

Choose a reason for hiding this comment

Uh oh!

feeblefakie left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Deprecate `include-metadata` option for data loader export #3159

Deprecate `include-metadata` option for data loader export #3159

ypeckstadt commented Nov 13, 2025 •

edited

Loading