diff --git a/PLANS.md b/PLANS.md index 7d8044a9f71..1187cefb7b5 100644 --- a/PLANS.md +++ b/PLANS.md @@ -1,83 +1,83 @@ # Codex Execution Plans (ExecPlans): - + This document describes the requirements for an execution plan ("ExecPlan"), a design document that a coding agent can follow to deliver a working feature or system change. Treat the reader as a complete beginner to this repository: they have only the current working tree and the single ExecPlan file you provide. There is no memory of prior plans and no external context. - + ## How to use ExecPlans and PLANS.md - + When authoring an executable specification (ExecPlan), follow PLANS.md _to the letter_. If it is not in your context, refresh your memory by reading the entire PLANS.md file. Be thorough in reading (and re-reading) source material to produce an accurate specification. When creating a spec, start from the skeleton and flesh it out as you do your research. - -When implementing an executable specification (ExecPlan), do not prompt the user for "next steps"; simply proceed to the next milestone. Keep all sections up to date, add or split entries in the list at every stopping point to affirmatively state the progress made and next steps. Resolve ambiguities autonomously, and commit frequently. - + +When implementing an executable specification (ExecPlan), do not prompt the user for "next steps"; always proceed to the next milestone. Keep all sections up to date, add or split entries in the list at every stopping point to affirmatively state the progress made and next steps. Resolve ambiguities autonomously and commit frequently. + When discussing an executable specification (ExecPlan), record decisions in a log in the spec for posterity; it should be unambiguously clear why any change to the specification was made. ExecPlans are living documents, and it should always be possible to restart from _only_ the ExecPlan and no other work. - -When researching a design with challenging requirements or significant unknowns, use milestones to implement proof of concepts, "toy implementations", etc., that allow validating whether the user's proposal is feasible. Read the source code of libraries by finding or acquiring them, research deeply, and include prototypes to guide a fuller implementation. - + +When researching a design with challenging requirements or significant unknowns, use milestones to implement proof of concepts, "toy implementations," etc., that allow validating whether the user's proposal is feasible. Read the source code of libraries by finding or acquiring them, research deeply, and include prototypes to guide a fuller implementation. + ## Requirements - + NON-NEGOTIABLE REQUIREMENTS: - + * Every ExecPlan must be fully self-contained. Self-contained means that in its current form it contains all knowledge and instructions needed for a novice to succeed. * Every ExecPlan is a living document. Contributors are required to revise it as progress is made, as discoveries occur, and as design decisions are finalized. Each revision must remain fully self-contained. * Every ExecPlan must enable a complete novice to implement the feature end-to-end without prior knowledge of this repo. -* Every ExecPlan must produce a demonstrably working behavior, not merely code changes to "meet a definition". +* Every ExecPlan must produce a demonstrably working behavior, not merely code changes to "meet a definition." * Every ExecPlan must define every term of art in plain language or do not use it. - + Purpose and intent come first. Begin by explaining, in a few sentences, why the work matters from a user's perspective: what someone can do after this change that they could not do before, and how to see it working. Then guide the reader through the exact steps to achieve that outcome, including what to edit, what to run, and what they should observe. - + The agent executing your plan can list files, read files, search, run the project, and run tests. It does not know any prior context and cannot infer what you meant from earlier milestones. Repeat any assumption you rely on. Do not point to external blogs or docs; if knowledge is required, embed it in the plan itself in your own words. If an ExecPlan builds upon a prior ExecPlan and that file is checked in, incorporate it by reference. If it is not, you must include all relevant context from that plan. - + ## Formatting - -Format and envelope are simple and strict. Each ExecPlan must be one single fenced code block labeled as `md` that begins and ends with triple backticks. Do not nest additional triple-backtick code fences inside; when you need to show commands, transcripts, diffs, or code, present them as indented blocks within that single fence. Use indentation for clarity rather than code fences inside an ExecPlan to avoid prematurely closing the ExecPlan's code fence. Use two newlines after every heading, use # and ## and so on, and correct syntax for ordered and unordered lists. - + +Format and envelope are straightforward and strict. Each ExecPlan must be one single fenced code block labeled as `md` that begins and ends with triple backticks. Do not nest additional triple-backtick code fences inside; when you need to show commands, transcripts, diffs, or code, present them as indented blocks within that single fence. Use indentation for clarity rather than code fences inside an ExecPlan to avoid prematurely closing the ExecPlan's code fence. Use two newlines after every heading, use # and ## and so on, and correct syntax for ordered and unordered lists. + When writing an ExecPlan to a Markdown (.md) file where the content of the file *is only* the single ExecPlan, you should omit the triple backticks. - -Write in plain prose. Prefer sentences over lists. Avoid checklists, tables, and long enumerations unless brevity would obscure meaning. Checklists are permitted only in the `Progress` section, where they are mandatory. Narrative sections must remain prose-first. - + +Write in plain prose. Prefer sentences to lists. Avoid checklists, tables, and long enumerations unless brevity would obscure meaning. Checklists are permitted only in the `Progress` section, where they are mandatory. Narrative sections must remain prose-first. + ## Guidelines - -Self-containment and plain language are paramount. If you introduce a phrase that is not ordinary English ("daemon", "middleware", "RPC gateway", "filter graph"), define it immediately and remind the reader how it manifests in this repository (for example, by naming the files or commands where it appears). Do not say "as defined previously" or "according to the architecture doc." Include the needed explanation here, even if you repeat yourself. - + +Self-containment and plain language are paramount. If you introduce a phrase that is not ordinary English ("daemon," "middleware," "RPC gateway," "filter graph"), define it immediately and remind the reader how it manifests in this repository (for example, by naming the files or commands where it appears). Do not say "as defined previously" or "according to the architecture doc." Include the necessary explanation here, even if you repeat yourself. + Avoid common failure modes. Do not rely on undefined jargon. Do not describe "the letter of a feature" so narrowly that the resulting code compiles but does nothing meaningful. Do not outsource key decisions to the reader. When ambiguity exists, resolve it in the plan itself and explain why you chose that path. Err on the side of over-explaining user-visible effects and under-specifying incidental implementation details. - -Anchor the plan with observable outcomes. State what the user can do after implementation, the commands to run, and the outputs they should see. Acceptance should be phrased as behavior a human can verify ("after starting the server, navigating to [http://localhost:8080/health](http://localhost:8080/health) returns HTTP 200 with body OK") rather than internal attributes ("added a HealthCheck struct"). If a change is internal, explain how its impact can still be demonstrated (for example, by running tests that fail before and pass after, and by showing a scenario that uses the new behavior). - -Specify repository context explicitly. Name files with full repository-relative paths, name functions and modules precisely, and describe where new files should be created. If touching multiple areas, include a short orientation paragraph that explains how those parts fit together so a novice can navigate confidently. When running commands, show the working directory and exact command line. When outcomes depend on environment, state the assumptions and provide alternatives when reasonable. - + +Anchor the plan with observable outcomes. State what the user can do after implementation, the commands to run, and the outputs they should see. Acceptance should be phrased as behavior a human can verify ("after starting the server, navigating to [http://localhost:8080/health](http://localhost:8080/health) returns HTTP 200 with body OK") rather than internal attributes ("added a health check struct"). If a change is internal, explain how its impact can still be demonstrated (for example, by running tests that fail before and pass after, and by showing a scenario that uses the new behavior). + +Specify the repository context explicitly. Name files with full repository-relative paths, name functions, and modules precisely, and describe where new files should be created. If touching multiple areas, include a short orientation paragraph that explains how those parts fit together so a novice can navigate confidently. When running commands, show the working directory and exact command line. When outcomes depend on environment, state the assumptions and provide alternatives when reasonable. + Be idempotent and safe. Write the steps so they can be run multiple times without causing damage or drift. If a step can fail halfway, include how to retry or adapt. If a migration or destructive operation is necessary, spell out backups or safe fallbacks. Prefer additive, testable changes that can be validated as you go. - + Validation is not optional. Include instructions to run tests, to start the system if applicable, and to observe it doing something useful. Describe comprehensive testing for any new features or capabilities. Include expected outputs and error messages so a novice can tell success from failure. Where possible, show how to prove that the change is effective beyond compilation (for example, through a small end-to-end scenario, a CLI invocation, or an HTTP request/response transcript). State the exact test commands appropriate to the project’s toolchain and how to interpret their results. - + Capture evidence. When your steps produce terminal output, short diffs, or logs, include them inside the single fenced block as indented examples. Keep them concise and focused on what proves success. If you need to include a patch, prefer file-scoped diffs or small excerpts that a reader can recreate by following your instructions rather than pasting large blobs. - + ## Milestones - + Milestones are narrative, not bureaucracy. If you break the work into milestones, introduce each with a brief paragraph that describes the scope, what will exist at the end of the milestone that did not exist before, the commands to run, and the acceptance you expect to observe. Keep it readable as a story: goal, work, result, proof. Progress and milestones are distinct: milestones tell the story, progress tracks granular work. Both must exist. Never abbreviate a milestone merely for the sake of brevity, do not leave out details that could be crucial to a future implementation. - + Each milestone must be independently verifiable and incrementally implement the overall goal of the execution plan. - + ## Living plans and design decisions - + * ExecPlans are living documents. As you make key design decisions, update the plan to record both the decision and the thinking behind it. Record all decisions in the `Decision Log` section. * ExecPlans must contain and maintain a `Progress` section, a `Surprises & Discoveries` section, a `Decision Log`, and an `Outcomes & Retrospective` section. These are not optional. * When you discover optimizer behavior, performance tradeoffs, unexpected bugs, or inverse/unapply semantics that shaped your approach, capture those observations in the `Surprises & Discoveries` section with short evidence snippets (test output is ideal). * If you change course mid-implementation, document why in the `Decision Log` and reflect the implications in `Progress`. Plans are guides for the next contributor as much as checklists for you. * At completion of a major task or the full plan, write an `Outcomes & Retrospective` entry summarizing what was achieved, what remains, and lessons learned. - + # Prototyping milestones and parallel implementations - + It is acceptable—-and often encouraged—-to include explicit prototyping milestones when they de-risk a larger change. Examples: adding a low-level operator to a dependency to validate feasibility, or exploring two composition orders while measuring optimizer effects. Keep prototypes additive and testable. Clearly label the scope as “prototyping”; describe how to run and observe results; and state the criteria for promoting or discarding the prototype. - + Prefer additive code changes followed by subtractions that keep tests passing. Parallel implementations (e.g., keeping an adapter alongside an older path during migration) are fine when they reduce risk or enable tests to continue passing during a large migration. Describe how to validate both paths and how to retire one safely with tests. When working with multiple new libraries or feature areas, consider creating spikes that evaluate the feasibility of these features _independently_ of one another, proving that the external library performs as expected and implements the features we need in isolation. - + ## Skeleton of a Good ExecPlan - + ```md # This ExecPlan is a living document. The sections `Progress`, `Surprises & Discoveries`, `Decision Log`, and `Outcomes & Retrospective` must be kept up to date as work proceeds. -If PLANS.md file is checked into the repo, reference the path to that file here from the repository root and note that this document must be maintained in accordance with PLANS.md. +If the PLANS.md file is checked into the repo, reference the path to that file here from the repository root and note that this document must be maintained in accordance with PLANS.md. ## Purpose / Big Picture @@ -146,7 +146,7 @@ In crates/foo/planner.rs, define: fn plan(&self, observed: &Observed) -> Vec; } ``` - + If you follow the guidance above, a single, stateless agent -- or a human novice -- can read your ExecPlan from top to bottom and produce a working, observable result. That is the bar: SELF-CONTAINED, SELF-SUFFICIENT, NOVICE-GUIDING, OUTCOME-FOCUSED. - -When you revise a plan, you must ensure your changes are comprehensively reflected across all sections, including the living document sections, and you must write a note at the bottom of the plan describing the change and the reason why. ExecPlans must describe not just the what but the why for almost everything. + +When you revise a plan, you must ensure your changes are comprehensively reflected across all sections, including the living document sections. You must write a note at the bottom of the plan describing the change and the reason why. ExecPlans must describe not just what but why for almost everything. diff --git a/core/common/io/src/main/java/org/eclipse/rdf4j/common/io/NioFile.java b/core/common/io/src/main/java/org/eclipse/rdf4j/common/io/NioFile.java index b14e45d37d2..fd69be5a2f4 100644 --- a/core/common/io/src/main/java/org/eclipse/rdf4j/common/io/NioFile.java +++ b/core/common/io/src/main/java/org/eclipse/rdf4j/common/io/NioFile.java @@ -56,6 +56,29 @@ public final class NioFile implements Closeable { private volatile boolean explictlyClosed; + /** + * Optional factory used to create FileChannel instances, primarily for testing where a delegating channel can + * simulate failures. If not set, {@link FileChannel#open(Path, java.nio.file.OpenOption...)} is used directly. + */ + private static volatile ChannelFactory channelFactory; + + /** + * Functional interface for creating FileChannel instances. Intended for test injection. + */ + @FunctionalInterface + public interface ChannelFactory { + FileChannel open(Path path, Set options) throws IOException; + } + + /** + * Install a factory that will be used to create FileChannel instances. Intended for tests only. + * + * Passing {@code null} restores the default behavior. + */ + public static void setChannelFactoryForTesting(ChannelFactory factory) { + channelFactory = factory; + } + /** * Constructor Opens a file in read/write mode, creating a new one if the file doesn't exist. * @@ -110,7 +133,12 @@ private static Set toOpenOptions(String mode) { * @throws IOException */ private void open() throws IOException { - fc = FileChannel.open(file.toPath(), openOptions); + ChannelFactory factory = channelFactory; + if (factory != null) { + fc = factory.open(file.toPath(), openOptions); + } else { + fc = FileChannel.open(file.toPath(), openOptions); + } } /** @@ -423,4 +451,5 @@ public int readInt(long offset) throws IOException { } return buf.getInt(0); } + } diff --git a/core/model-vocabulary/src/main/java/org/eclipse/rdf4j/model/vocabulary/CONFIG.java b/core/model-vocabulary/src/main/java/org/eclipse/rdf4j/model/vocabulary/CONFIG.java index 852467d0d41..bc9096e4386 100644 --- a/core/model-vocabulary/src/main/java/org/eclipse/rdf4j/model/vocabulary/CONFIG.java +++ b/core/model-vocabulary/src/main/java/org/eclipse/rdf4j/model/vocabulary/CONFIG.java @@ -213,7 +213,7 @@ public static final class Sail { public final static IRI impl = createIRI(NAMESPACE, "sail.impl"); /** - * tag:rdf4j.org,2023:config/sail.iterationCacheSyncTreshold + * tag:rdf4j.org,2023:config/sail.iterationCacheSyncThreshold */ public final static IRI iterationCacheSyncThreshold = createIRI(NAMESPACE, "sail.iterationCacheSyncThreshold"); @@ -276,6 +276,28 @@ public static final class Native { * tag:rdf4j.org,2023:config/native.namespaceIDCacheSize */ public final static IRI namespaceIDCacheSize = createIRI(NAMESPACE, "native.namespaceIDCacheSize"); + + // ValueStore WAL configuration properties + /** tag:rdf4j.org,2023:config/native.walMaxSegmentBytes */ + public final static IRI walMaxSegmentBytes = createIRI(NAMESPACE, "native.walMaxSegmentBytes"); + /** tag:rdf4j.org,2023:config/native.walQueueCapacity */ + public final static IRI walQueueCapacity = createIRI(NAMESPACE, "native.walQueueCapacity"); + /** tag:rdf4j.org,2023:config/native.walBatchBufferBytes */ + public final static IRI walBatchBufferBytes = createIRI(NAMESPACE, "native.walBatchBufferBytes"); + /** tag:rdf4j.org,2023:config/native.walSyncPolicy */ + public final static IRI walSyncPolicy = createIRI(NAMESPACE, "native.walSyncPolicy"); + /** tag:rdf4j.org,2023:config/native.walSyncIntervalMillis */ + public final static IRI walSyncIntervalMillis = createIRI(NAMESPACE, "native.walSyncIntervalMillis"); + /** tag:rdf4j.org,2023:config/native.walIdlePollIntervalMillis */ + public final static IRI walIdlePollIntervalMillis = createIRI(NAMESPACE, "native.walIdlePollIntervalMillis"); + /** tag:rdf4j.org,2023:config/native.walDirectoryName */ + public final static IRI walDirectoryName = createIRI(NAMESPACE, "native.walDirectoryName"); + /** tag:rdf4j.org,2023:config/native.walSyncBootstrapOnOpen */ + public final static IRI walSyncBootstrapOnOpen = createIRI(NAMESPACE, "native.walSyncBootstrapOnOpen"); + /** tag:rdf4j.org,2023:config/native.walAutoRecoverOnOpen */ + public final static IRI walAutoRecoverOnOpen = createIRI(NAMESPACE, "native.walAutoRecoverOnOpen"); + /** tag:rdf4j.org,2025:config/native.walEnabled */ + public final static IRI walEnabled = createIRI(NAMESPACE, "native.walEnabled"); } /** diff --git a/core/model/src/main/java/org/eclipse/rdf4j/model/util/Configurations.java b/core/model/src/main/java/org/eclipse/rdf4j/model/util/Configurations.java index 7c9bb003ea2..1679b40c103 100644 --- a/core/model/src/main/java/org/eclipse/rdf4j/model/util/Configurations.java +++ b/core/model/src/main/java/org/eclipse/rdf4j/model/util/Configurations.java @@ -12,6 +12,7 @@ package org.eclipse.rdf4j.model.util; import java.util.HashSet; +import java.util.Objects; import java.util.Optional; import java.util.Set; @@ -64,7 +65,7 @@ public static boolean hasLegacyConfiguration(Model configModel) { /** * Retrieve a property value for the supplied subject as a {@link Resource} if present, falling back to a supplied - * legacy property . + * legacy property. *

* This method allows querying repository config models with a mix of old and new namespaces. * @@ -72,7 +73,7 @@ public static boolean hasLegacyConfiguration(Model configModel) { * @param subject the subject of the property. * @param property the property to retrieve the value of. * @param legacyProperty legacy property to use if the supplied property has no value in the model. - * @return the resource value for supplied subject and property (or the legacy property ), if present. + * @return the resource value for supplied subject and property (or the legacy property), if present. */ @InternalUseOnly public static Optional getResourceValue(Model model, Resource subject, IRI property, IRI legacyProperty) { @@ -92,7 +93,7 @@ public static Optional getResourceValue(Model model, Resource subject, /** * Retrieve a property value for the supplied subject as a {@link Literal} if present, falling back to a supplied - * legacy property . + * legacy property. *

* This method allows querying repository config models with a mix of old and new namespaces. * @@ -100,10 +101,14 @@ public static Optional getResourceValue(Model model, Resource subject, * @param subject the subject of the property. * @param property the property to retrieve the value of. * @param legacyProperty legacy property to use if the supplied property has no value in the model. - * @return the literal value for supplied subject and property (or the legacy property ), if present. + * @return the literal value for the supplied subject and property (or the legacy property), if present. */ @InternalUseOnly public static Optional getLiteralValue(Model model, Resource subject, IRI property, IRI legacyProperty) { + Objects.requireNonNull(model, "model must not be null"); + Objects.requireNonNull(subject, "subject must not be null"); + Objects.requireNonNull(property, "property must not be null"); + Objects.requireNonNull(legacyProperty, "legacyProperty must not be null"); var preferredProperty = useLegacyConfig() ? legacyProperty : property; var fallbackProperty = useLegacyConfig() ? property : legacyProperty; @@ -117,9 +122,27 @@ public static Optional getLiteralValue(Model model, Resource subject, I return fallbackResult; } + /** + * Retrieve a property value for the supplied subject as a {@link Literal} if present. + *

+ * + * @param model the model to retrieve property values from. + * @param subject the subject of the property. + * @param property the property to retrieve the value of. + * @return the literal value for the supplied subject and property, if present. + */ + @InternalUseOnly + public static Optional getLiteralValue(Model model, Resource subject, IRI property) { + Objects.requireNonNull(model, "model must not be null"); + Objects.requireNonNull(subject, "subject must not be null"); + Objects.requireNonNull(property, "property must not be null"); + + return Models.objectLiteral(model.getStatements(subject, property, null)); + } + /** * Retrieve a property value for the supplied subject as a {@link Value} if present, falling back to a supplied - * legacy property . + * legacy property. *

* This method allows querying repository config models with a mix of old and new namespaces. * @@ -127,7 +150,7 @@ public static Optional getLiteralValue(Model model, Resource subject, I * @param subject the subject of the property. * @param property the property to retrieve the value of. * @param legacyProperty legacy property to use if the supplied property has no value in the model. - * @return the literal value for supplied subject and property (or the legacy property ), if present. + * @return the literal value for supplied subject and property (or the legacy property), if present. */ @InternalUseOnly public static Optional getValue(Model model, Resource subject, IRI property, IRI legacyProperty) { @@ -197,7 +220,7 @@ public static Set getPropertyValues(Model model, Resource subject, IRI pr * @param subject the subject of the property. * @param property the property to retrieve the value of. * @param legacyProperty legacy property to use if the supplied property has no value in the model. - * @return the IRI value for supplied subject and property (or the legacy property ), if present. + * @return the IRI value for supplied subject and property (or the legacy property), if present. */ @InternalUseOnly public static Optional getIRIValue(Model model, Resource subject, IRI property, IRI legacyProperty) { diff --git a/core/sail/api/src/main/java/org/eclipse/rdf4j/sail/helpers/DirectoryLockManager.java b/core/sail/api/src/main/java/org/eclipse/rdf4j/sail/helpers/DirectoryLockManager.java index 7f1d45f98cb..0125b035848 100644 --- a/core/sail/api/src/main/java/org/eclipse/rdf4j/sail/helpers/DirectoryLockManager.java +++ b/core/sail/api/src/main/java/org/eclipse/rdf4j/sail/helpers/DirectoryLockManager.java @@ -11,6 +11,7 @@ package org.eclipse.rdf4j.sail.helpers; import java.io.BufferedReader; +import java.io.BufferedWriter; import java.io.File; import java.io.FileReader; import java.io.FileWriter; diff --git a/core/sail/nativerdf/pom.xml b/core/sail/nativerdf/pom.xml index 01152a79759..86d77a6f846 100644 --- a/core/sail/nativerdf/pom.xml +++ b/core/sail/nativerdf/pom.xml @@ -40,6 +40,10 @@ rdf4j-model ${project.version} + + com.fasterxml.jackson.core + jackson-core + org.slf4j slf4j-api diff --git a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/MemoryMappedTxnStatusFile.java b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/MemoryMappedTxnStatusFile.java new file mode 100644 index 00000000000..d112e32f83d --- /dev/null +++ b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/MemoryMappedTxnStatusFile.java @@ -0,0 +1,156 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf; + +import static java.nio.charset.StandardCharsets.US_ASCII; + +import java.io.File; +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.MappedByteBuffer; +import java.nio.channels.FileChannel; +import java.nio.file.Files; +import java.nio.file.StandardOpenOption; +import java.util.EnumSet; + +import org.eclipse.rdf4j.common.annotation.Experimental; + +/** + * Writes transaction statuses to a memory-mapped file. Since the OS is responsible for flushing changes to disk, this + * is generally faster than using regular file I/O. If the JVM crashes, the last written status should still be intact, + * but the change will not be visible until the OS has flushed the page to disk. If the OS or DISK crashes, data may be + * lost or corrupted. Same for power loss. This can be mitigated by setting the {@link #ALWAYS_FORCE_SYNC_PROP} system + * property to true, which forces a sync to disk on every status change. + */ +@Experimental +class MemoryMappedTxnStatusFile extends TxnStatusFile { + + /** + * The name of the transaction status file. + */ + public static final String FILE_NAME = "txn-status"; + + /** + * We currently store a single status byte, but this constant makes it trivial to extend the layout later if needed. + */ + private static final int MAPPED_SIZE = 1; + + private static final String ALWAYS_FORCE_SYNC_PROP = "org.eclipse.rdf4j.sail.nativerdf.MemoryMappedTxnStatusFile.alwaysForceSync"; + + static boolean ALWAYS_FORCE_SYNC = Boolean.getBoolean(ALWAYS_FORCE_SYNC_PROP); + + private final File statusFile; + private final FileChannel channel; + private final MappedByteBuffer mapped; + + /** + * Creates a new transaction status file. New files are initialized with {@link TxnStatus#NONE}. + * + * @param dataDir The directory for the transaction status file. + * @throws IOException If the file could not be opened or created. + */ + public MemoryMappedTxnStatusFile(File dataDir) throws IOException { + super(); + this.statusFile = new File(dataDir, FILE_NAME); + + ALWAYS_FORCE_SYNC = !Boolean.getBoolean(ALWAYS_FORCE_SYNC_PROP); + + EnumSet openOptions = EnumSet.of(StandardOpenOption.READ, StandardOpenOption.WRITE, + StandardOpenOption.CREATE); + + this.channel = FileChannel.open(statusFile.toPath(), openOptions.toArray(new StandardOpenOption[0])); + + long size = channel.size(); + + // Ensure the file is at least MAPPED_SIZE bytes so we can map it safely. + // If it was previously empty, we treat that as NONE (which is also byte 0). + if (size < MAPPED_SIZE) { + channel.position(MAPPED_SIZE - 1); + int write = channel.write(ByteBuffer.wrap(TxnStatus.NONE.getOnDisk())); + if (write != 1) { + throw new IOException("Failed to initialize transaction status file"); + } + channel.force(true); + } + + this.mapped = channel.map(FileChannel.MapMode.READ_WRITE, 0, MAPPED_SIZE); + } + + public void close() throws IOException { + // We rely on the GC to eventually unmap the MappedByteBuffer; explicitly + // closing the channel is enough for our purposes here. + channel.close(); + } + + /** + * Writes the specified transaction status to file. + * + * @param txnStatus The transaction status to write. + * @param forceSync If true, forces a sync to disk after writing the status. + */ + public void setTxnStatus(TxnStatus txnStatus, boolean forceSync) { + if (disabled) { + return; + } + + mapped.put(0, txnStatus.getOnDisk()[0]); + if (ALWAYS_FORCE_SYNC || forceSync) { + mapped.force(); + } + } + + /** + * Reads the transaction status from file. + * + * @return The read transaction status, or {@link TxnStatus#UNKNOWN} when the file contains an unrecognized status + * string. + * @throws IOException If the transaction status file could not be read. + */ + public TxnStatus getTxnStatus() throws IOException { + if (disabled) { + return TxnStatus.NONE; + } + + try { + return statusMapping[mapped.get(0)]; + } catch (IndexOutOfBoundsException e) { + return getTxnStatusDeprecated(); + } + } + + private TxnStatus getTxnStatusDeprecated() throws IOException { + if (disabled) { + return TxnStatus.NONE; + } + + // Read the full file contents as a string, for compatibility with very old + // versions that stored the enum name instead of a bitfield. + byte[] bytes = Files.readAllBytes(statusFile.toPath()); + + if (bytes.length == 0) { + return TxnStatus.NONE; + } + + String s = new String(bytes, US_ASCII); + try { + return TxnStatus.valueOf(s); + } catch (IllegalArgumentException e) { + // use platform encoding for backwards compatibility with versions + // older than 2.6.6: + s = new String(bytes); + try { + return TxnStatus.valueOf(s); + } catch (IllegalArgumentException e2) { + return TxnStatus.UNKNOWN; + } + } + } +} diff --git a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/NativeSailStore.java b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/NativeSailStore.java index 1c88be4e601..cc84e1a08bb 100644 --- a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/NativeSailStore.java +++ b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/NativeSailStore.java @@ -12,6 +12,11 @@ import java.io.File; import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.StandardOpenOption; +import java.time.Duration; import java.util.ArrayList; import java.util.Collections; import java.util.LinkedHashSet; @@ -19,9 +24,12 @@ import java.util.Map; import java.util.Map.Entry; import java.util.Objects; +import java.util.OptionalLong; import java.util.Set; +import java.util.UUID; import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.locks.ReentrantLock; +import java.util.regex.Pattern; import org.eclipse.rdf4j.common.iteration.CloseableIteration; import org.eclipse.rdf4j.common.iteration.CloseableIteratorIteration; @@ -45,7 +53,10 @@ import org.eclipse.rdf4j.sail.base.SailSource; import org.eclipse.rdf4j.sail.base.SailStore; import org.eclipse.rdf4j.sail.nativerdf.btree.RecordIterator; +import org.eclipse.rdf4j.sail.nativerdf.datastore.DataStore; import org.eclipse.rdf4j.sail.nativerdf.model.NativeValue; +import org.eclipse.rdf4j.sail.nativerdf.wal.ValueStoreWAL; +import org.eclipse.rdf4j.sail.nativerdf.wal.ValueStoreWalConfig; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -57,14 +68,18 @@ class NativeSailStore implements SailStore { final Logger logger = LoggerFactory.getLogger(NativeSailStore.class); + private static final Pattern WAL_SEGMENT_PATTERN = Pattern.compile("wal-\\d+\\.v1(?:\\.gz)?"); private final TripleStore tripleStore; + private final ValueStoreWAL valueStoreWal; + private final ValueStore valueStore; private final NamespaceStore namespaceStore; private final ContextStore contextStore; + private final boolean walEnabled; /** * A lock to control concurrent access by {@link NativeSailSink} to the TripleStore, ValueStore, and NamespaceStore. @@ -83,29 +98,210 @@ class NativeSailStore implements SailStore { */ public NativeSailStore(File dataDir, String tripleIndexes) throws IOException, SailException { this(dataDir, tripleIndexes, false, ValueStore.VALUE_CACHE_SIZE, ValueStore.VALUE_ID_CACHE_SIZE, - ValueStore.NAMESPACE_CACHE_SIZE, ValueStore.NAMESPACE_ID_CACHE_SIZE); + ValueStore.NAMESPACE_CACHE_SIZE, ValueStore.NAMESPACE_ID_CACHE_SIZE, + -1L, -1, -1, null, -1L, -1L, null, false, false, true); } /** * Creates a new {@link NativeSailStore}. */ + + public NativeSailStore(File dataDir, String tripleIndexes, boolean forceSync, int valueCacheSize, + int valueIDCacheSize, int namespaceCacheSize, int namespaceIDCacheSize, long walMaxSegmentBytes, + int walQueueCapacity, int walBatchBufferBytes, + ValueStoreWalConfig.SyncPolicy walSyncPolicy, + long walSyncIntervalMillis, long walIdlePollIntervalMillis, String walDirectoryName) + throws IOException, SailException { + this(dataDir, tripleIndexes, forceSync, valueCacheSize, valueIDCacheSize, namespaceCacheSize, + namespaceIDCacheSize, walMaxSegmentBytes, walQueueCapacity, walBatchBufferBytes, walSyncPolicy, + walSyncIntervalMillis, walIdlePollIntervalMillis, walDirectoryName, false, false, true); + } + public NativeSailStore(File dataDir, String tripleIndexes, boolean forceSync, int valueCacheSize, - int valueIDCacheSize, int namespaceCacheSize, int namespaceIDCacheSize) throws IOException, SailException { + int valueIDCacheSize, int namespaceCacheSize, int namespaceIDCacheSize, long walMaxSegmentBytes, + int walQueueCapacity, int walBatchBufferBytes, + ValueStoreWalConfig.SyncPolicy walSyncPolicy, + long walSyncIntervalMillis, long walIdlePollIntervalMillis, String walDirectoryName, + boolean walSyncBootstrapOnOpen, boolean walAutoRecoverOnOpen, boolean walEnabled) + throws IOException, SailException { + this.walEnabled = walEnabled; + NamespaceStore createdNamespaceStore = null; + ValueStoreWAL createdWal = null; + ValueStore createdValueStore = null; + TripleStore createdTripleStore = null; + ContextStore createdContextStore = null; boolean initialized = false; try { - namespaceStore = new NamespaceStore(dataDir); - valueStore = new ValueStore(dataDir, forceSync, valueCacheSize, valueIDCacheSize, namespaceCacheSize, - namespaceIDCacheSize); - tripleStore = new TripleStore(dataDir, tripleIndexes, forceSync); - contextStore = new ContextStore(this, dataDir); + createdNamespaceStore = new NamespaceStore(dataDir); + Path walDir = dataDir.toPath() + .resolve(walDirectoryName != null && !walDirectoryName.isEmpty() ? walDirectoryName + : ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + boolean enableWal = shouldEnableWal(dataDir, walDir); + ValueStoreWalConfig walConfig = null; + if (enableWal) { + String storeUuid = loadOrCreateWalUuid(walDir); + ValueStoreWalConfig.Builder walBuilder = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid(storeUuid); + if (walMaxSegmentBytes > 0) { + walBuilder.maxSegmentBytes(walMaxSegmentBytes); + } + if (walQueueCapacity > 0) { + walBuilder.queueCapacity(walQueueCapacity); + } + if (walBatchBufferBytes > 0) { + walBuilder.batchBufferBytes(walBatchBufferBytes); + } + if (walSyncPolicy != null) { + walBuilder.syncPolicy(walSyncPolicy); + } + if (walSyncIntervalMillis >= 0) { + walBuilder.syncInterval(Duration.ofMillis(walSyncIntervalMillis)); + } + if (walIdlePollIntervalMillis >= 0) { + walBuilder.idlePollInterval(Duration.ofMillis(walIdlePollIntervalMillis)); + } + // propagate bootstrap mode + walBuilder.syncBootstrapOnOpen(walSyncBootstrapOnOpen); + walBuilder.recoverValueStoreOnOpen(walAutoRecoverOnOpen); + walConfig = walBuilder.build(); + createdWal = ValueStoreWAL.open(walConfig); + } else { + createdWal = null; + } + createdValueStore = new ValueStore(dataDir, forceSync, valueCacheSize, valueIDCacheSize, + namespaceCacheSize, namespaceIDCacheSize, createdWal); + createdTripleStore = new TripleStore(dataDir, tripleIndexes, forceSync); + + // Assign fields required by ContextStore before constructing it + namespaceStore = createdNamespaceStore; + valueStoreWal = createdWal; + valueStore = createdValueStore; + tripleStore = createdTripleStore; + + // Now ContextStore can safely read from this store + createdContextStore = new ContextStore(this, dataDir); initialized = true; } finally { if (!initialized) { - close(); + closeQuietly(createdContextStore); + closeQuietly(createdTripleStore); + closeQuietly(createdValueStore); + closeQuietly(createdWal); + closeQuietly(createdNamespaceStore); + } + } + // Finalize assignment of contextStore + contextStore = createdContextStore; + } + + private String loadOrCreateWalUuid(Path walDir) throws IOException { + Files.createDirectories(walDir); + Path file = walDir.resolve("store.uuid"); + if (Files.exists(file)) { + return Files.readString(file, StandardCharsets.UTF_8).trim(); + } + String uuid = UUID.randomUUID().toString(); + Files.writeString(file, uuid, StandardCharsets.UTF_8, StandardOpenOption.CREATE, + StandardOpenOption.TRUNCATE_EXISTING); + return uuid; + } + + private boolean shouldEnableWal(File dataDir, Path walDir) throws IOException { + if (!walEnabled) { + if (logger.isDebugEnabled()) { + if (hasExistingWalSegments(walDir)) { + logger.debug( + "ValueStore WAL is disabled via configuration but {} contains WAL segments; ignoring them.", + walDir); + } else { + logger.debug("ValueStore WAL disabled via configuration for {}", dataDir); + } + } + + return false; + } + // Respect read-only data directories: do not enable WAL when we can't write + if (!dataDir.canWrite()) { + return false; + } + if (hasExistingWalSegments(walDir)) { +// writeBootstrapMarker(walDir, "enabled-existing-wal"); + return true; + } + try (DataStore values = new DataStore(dataDir, "values", false)) { + if (values.getMaxID() > 0) { +// writeBootstrapMarker(walDir, "enabled-rebuild-existing-values"); + return true; + } + } +// writeBootstrapMarker(walDir, "enabled-empty-store"); + return true; + } + + private boolean hasExistingWalSegments(Path walDir) throws IOException { + if (!Files.isDirectory(walDir)) { + return false; + } + try (var stream = Files.list(walDir)) { + return stream.anyMatch(path -> WAL_SEGMENT_PATTERN.matcher(path.getFileName().toString()).matches()); + } + } + + private void writeBootstrapMarker(Path walDir, String state) { + try { + Files.createDirectories(walDir); + Path marker = walDir.resolve("bootstrap.info"); + String content = "state=" + state + "\n"; + Files.writeString(marker, content, StandardCharsets.UTF_8, StandardOpenOption.CREATE, + StandardOpenOption.TRUNCATE_EXISTING); + } catch (IOException e) { + logger.warn("Failed to write WAL bootstrap marker", e); + } + } + + private void closeQuietly(ContextStore store) { + if (store != null) { + store.close(); + } + } + + private void closeQuietly(TripleStore store) { + if (store != null) { + try { + store.close(); + } catch (IOException e) { + logger.warn("Failed to close triple store", e); } } } + private void closeQuietly(ValueStore store) { + if (store != null) { + try { + store.close(); + } catch (IOException e) { + logger.warn("Failed to close value store", e); + } + } + } + + private void closeQuietly(ValueStoreWAL wal) { + if (wal != null) { + try { + wal.close(); + } catch (IOException e) { + logger.warn("Failed to close value store WAL", e); + } + } + } + + private void closeQuietly(NamespaceStore store) { + if (store != null) { + store.close(); + } + } + @Override public ValueFactory getValueFactory() { return valueStore; @@ -129,8 +325,14 @@ public void close() throws SailException { valueStore.close(); } } finally { - if (tripleStore != null) { - tripleStore.close(); + try { + if (valueStoreWal != null) { + valueStoreWal.close(); + } + } finally { + if (tripleStore != null) { + tripleStore.close(); + } } } } @@ -353,11 +555,22 @@ public NativeSailSink(boolean explicit) throws SailException { this.explicit = explicit; } + private long walHighWaterMark = ValueStoreWAL.NO_LSN; + @Override public void close() { // no-op } + private int storeValueId(Value value) throws IOException { + int id = valueStore.storeValue(value); + OptionalLong walLsn = valueStore.drainPendingWalHighWaterMark(); + if (walLsn.isPresent()) { + walHighWaterMark = Math.max(walHighWaterMark, walLsn.getAsLong()); + } + return id; + } + @Override public void prepare() throws SailException { // serializable is not supported at this level @@ -368,6 +581,10 @@ public synchronized void flush() throws SailException { sinkStoreAccessLock.lock(); try { try { + if (walHighWaterMark > ValueStoreWAL.NO_LSN) { + valueStore.awaitWalDurable(walHighWaterMark); + walHighWaterMark = ValueStoreWAL.NO_LSN; + } valueStore.sync(); } finally { try { @@ -472,13 +689,13 @@ public void approveAll(Set approved, Set approvedContexts) Value obj = statement.getObject(); Resource context = statement.getContext(); - int subjID = valueStore.storeValue(subj); - int predID = valueStore.storeValue(pred); - int objID = valueStore.storeValue(obj); + int subjID = storeValueId(subj); + int predID = storeValueId(pred); + int objID = storeValueId(obj); int contextID = 0; if (context != null) { - contextID = valueStore.storeValue(context); + contextID = storeValueId(context); } boolean wasNew = tripleStore.storeTriple(subjID, predID, objID, contextID, explicit); @@ -532,9 +749,9 @@ private boolean addStatement(Resource subj, IRI pred, Value obj, boolean explici sinkStoreAccessLock.lock(); try { startTriplestoreTransaction(); - int subjID = valueStore.storeValue(subj); - int predID = valueStore.storeValue(pred); - int objID = valueStore.storeValue(obj); + int subjID = storeValueId(subj); + int predID = storeValueId(pred); + int objID = storeValueId(obj); if (contexts.length == 0) { contexts = new Resource[] { null }; @@ -543,7 +760,7 @@ private boolean addStatement(Resource subj, IRI pred, Value obj, boolean explici for (Resource context : contexts) { int contextID = 0; if (context != null) { - contextID = valueStore.storeValue(context); + contextID = storeValueId(context); } boolean wasNew = tripleStore.storeTriple(subjID, predID, objID, contextID, explicit); diff --git a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/NativeStatementIterator.java b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/NativeStatementIterator.java index 29b803e6cb5..819d890965e 100644 --- a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/NativeStatementIterator.java +++ b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/NativeStatementIterator.java @@ -13,9 +13,10 @@ import static org.eclipse.rdf4j.sail.nativerdf.NativeStore.SOFT_FAIL_ON_CORRUPT_DATA_AND_REPAIR_INDEXES; import java.io.IOException; +import java.util.NoSuchElementException; import org.eclipse.rdf4j.common.io.ByteArrayUtil; -import org.eclipse.rdf4j.common.iteration.LookAheadIteration; +import org.eclipse.rdf4j.common.iteration.CloseableIteration; import org.eclipse.rdf4j.model.IRI; import org.eclipse.rdf4j.model.Resource; import org.eclipse.rdf4j.model.Statement; @@ -32,35 +33,21 @@ * A statement iterator that wraps a RecordIterator containing statement records and translates these records to * {@link Statement} objects. */ -class NativeStatementIterator extends LookAheadIteration { +class NativeStatementIterator implements CloseableIteration { private static final Logger logger = LoggerFactory.getLogger(NativeStatementIterator.class); - /*-----------* - * Variables * - *-----------*/ - private final RecordIterator btreeIter; - private final ValueStore valueStore; - /*--------------* - * Constructors * - *--------------*/ + private Statement nextElement; + private boolean closed = false; - /** - * Creates a new NativeStatementIterator. - */ public NativeStatementIterator(RecordIterator btreeIter, ValueStore valueStore) { this.btreeIter = btreeIter; this.valueStore = valueStore; } - /*---------* - * Methods * - *---------*/ - - @Override public Statement getNextElement() throws SailException { try { byte[] nextValue; @@ -107,7 +94,6 @@ public Statement getNextElement() throws SailException { } } - @Override protected void handleClose() throws SailException { try { btreeIter.close(); @@ -119,4 +105,79 @@ protected void handleClose() throws SailException { protected SailException causeIOException(IOException e) { return new SailException(e); } + + @Override + public final boolean hasNext() { + if (isClosed()) { + return false; + } + + try { + return lookAhead() != null; + } catch (NoSuchElementException logged) { + // The lookAhead() method shouldn't throw a NoSuchElementException since it should return null when there + // are no more elements. + logger.trace("LookAheadIteration threw NoSuchElementException:", logged); + return false; + } + } + + @Override + public final Statement next() { + if (isClosed()) { + throw new NoSuchElementException("The iteration has been closed."); + } + Statement result = lookAhead(); + + if (result != null) { + nextElement = null; + return result; + } else { + throw new NoSuchElementException(); + } + } + + /** + * Fetches the next element if it hasn't been fetched yet and stores it in {@link #nextElement}. + * + * @return The next element, or null if there are no more results. + */ + private Statement lookAhead() { + if (nextElement == null) { + nextElement = getNextElement(); + + if (nextElement == null) { + close(); + } + } + return nextElement; + } + + /** + * Throws an {@link UnsupportedOperationException}. + */ + @Override + public void remove() { + throw new UnsupportedOperationException(); + } + + /** + * Checks whether this CloseableIteration has been closed. + * + * @return true if the CloseableIteration has been closed, false otherwise. + */ + public final boolean isClosed() { + return closed; + } + + /** + * Calls {@link #handleClose()} upon first call and makes sure the resource closures are only executed once. + */ + @Override + public final void close() { + if (!closed) { + closed = true; + handleClose(); + } + } } diff --git a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/NativeStore.java b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/NativeStore.java index 992154b76ac..6b3c77d94fa 100644 --- a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/NativeStore.java +++ b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/NativeStore.java @@ -24,6 +24,7 @@ import org.apache.commons.io.FileUtils; import org.eclipse.rdf4j.collection.factory.api.CollectionFactory; import org.eclipse.rdf4j.collection.factory.mapdb.MapDb3CollectionFactory; +import org.eclipse.rdf4j.common.annotation.Experimental; import org.eclipse.rdf4j.common.annotation.InternalUseOnly; import org.eclipse.rdf4j.common.concurrent.locks.Lock; import org.eclipse.rdf4j.common.concurrent.locks.LockManager; @@ -45,6 +46,7 @@ import org.eclipse.rdf4j.sail.base.SnapshotSailStore; import org.eclipse.rdf4j.sail.helpers.AbstractNotifyingSail; import org.eclipse.rdf4j.sail.helpers.DirectoryLockManager; +import org.eclipse.rdf4j.sail.nativerdf.wal.ValueStoreWalConfig; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -182,6 +184,18 @@ protected SailStore createSailStore(File dataDir) throws IOException, SailExcept */ private final LockManager disabledIsolationLockManager = new LockManager(debugEnabled()); + // Optional WAL configuration propagated into NativeSailStore + private long walMaxSegmentBytes = -1L; + private int walQueueCapacity = -1; + private int walBatchBufferBytes = -1; + private ValueStoreWalConfig.SyncPolicy walSyncPolicy = null; + private long walSyncIntervalMillis = -1L; + private long walIdlePollIntervalMillis = -1L; + private String walDirectoryName = null; + private boolean walSyncBootstrapOnOpen = false; + private boolean walAutoRecoverOnOpen = false; + private boolean walEnabled = true; + /*--------------* * Constructors * *--------------*/ @@ -262,6 +276,109 @@ public void setNamespaceIDCacheSize(int namespaceIDCacheSize) { this.namespaceIDCacheSize = namespaceIDCacheSize; } + @Experimental + public void setWalMaxSegmentBytes(long walMaxSegmentBytes) { + this.walMaxSegmentBytes = walMaxSegmentBytes; + } + + @Experimental + public long getWalMaxSegmentBytes() { + return walMaxSegmentBytes; + } + + @Experimental + public void setWalQueueCapacity(int walQueueCapacity) { + this.walQueueCapacity = walQueueCapacity; + } + + @Experimental + public int getWalQueueCapacity() { + return walQueueCapacity; + } + + @Experimental + public void setWalBatchBufferBytes(int walBatchBufferBytes) { + this.walBatchBufferBytes = walBatchBufferBytes; + } + + @Experimental + public int getWalBatchBufferBytes() { + return walBatchBufferBytes; + } + + @Experimental + public void setWalSyncPolicy(ValueStoreWalConfig.SyncPolicy walSyncPolicy) { + this.walSyncPolicy = walSyncPolicy; + } + + @Experimental + public ValueStoreWalConfig.SyncPolicy getWalSyncPolicy() { + return walSyncPolicy; + } + + @Experimental + public void setWalSyncIntervalMillis(long walSyncIntervalMillis) { + this.walSyncIntervalMillis = walSyncIntervalMillis; + } + + @Experimental + public long getWalSyncIntervalMillis() { + return walSyncIntervalMillis; + } + + @Experimental + public void setWalIdlePollIntervalMillis(long walIdlePollIntervalMillis) { + this.walIdlePollIntervalMillis = walIdlePollIntervalMillis; + } + + @Experimental + public long getWalIdlePollIntervalMillis() { + return walIdlePollIntervalMillis; + } + + @Experimental + public void setWalDirectoryName(String walDirectoryName) { + this.walDirectoryName = walDirectoryName; + } + + @Experimental + public String getWalDirectoryName() { + return walDirectoryName; + } + + /** Ensure WAL bootstrap is synchronous during open (before new values are added). */ + @Experimental + public void setWalSyncBootstrapOnOpen(boolean walSyncBootstrapOnOpen) { + this.walSyncBootstrapOnOpen = walSyncBootstrapOnOpen; + } + + @Experimental + public boolean isWalSyncBootstrapOnOpen() { + return walSyncBootstrapOnOpen; + } + + /** Enable automatic ValueStore recovery from WAL during open. */ + @Experimental + public void setWalAutoRecoverOnOpen(boolean walAutoRecoverOnOpen) { + this.walAutoRecoverOnOpen = walAutoRecoverOnOpen; + } + + @Experimental + public boolean isWalAutoRecoverOnOpen() { + return walAutoRecoverOnOpen; + } + + /** Enable or disable the ValueStore WAL entirely. */ + @Experimental + public void setWalEnabled(boolean walEnabled) { + this.walEnabled = walEnabled; + } + + @Experimental + public boolean isWalEnabled() { + return walEnabled; + } + /** * @return Returns the {@link EvaluationStrategy}. */ @@ -346,16 +463,37 @@ protected void initializeInternal() throws SailException { try { Path versionPath = new File(dataDir, "nativerdf.ver").toPath(); - String version = versionPath.toFile().exists() ? Files.readString(versionPath, StandardCharsets.UTF_8) - : null; + String version; + try { + version = Files.readString(versionPath, StandardCharsets.UTF_8); + } catch (Exception e) { + version = null; + } + if (!VERSION.equals(version) && upgradeStore(dataDir, version)) { logger.debug("Data store upgraded to version " + VERSION); Files.writeString(versionPath, VERSION, StandardCharsets.UTF_8, StandardOpenOption.CREATE, StandardOpenOption.WRITE, StandardOpenOption.TRUNCATE_EXISTING); } - final NativeSailStore mainStore = new NativeSailStore(dataDir, tripleIndexes, forceSync, valueCacheSize, - valueIDCacheSize, namespaceCacheSize, namespaceIDCacheSize); - this.store = new SnapshotSailStore(mainStore, () -> new MemoryOverflowIntoNativeStore()) { + final NativeSailStore mainStore = new NativeSailStore( + dataDir, + tripleIndexes, + forceSync, + valueCacheSize, + valueIDCacheSize, + namespaceCacheSize, + namespaceIDCacheSize, + walMaxSegmentBytes, + walQueueCapacity, + walBatchBufferBytes, + walSyncPolicy, + walSyncIntervalMillis, + walIdlePollIntervalMillis, + walDirectoryName, + walSyncBootstrapOnOpen, + walAutoRecoverOnOpen, + walEnabled); + this.store = new SnapshotSailStore(mainStore, MemoryOverflowIntoNativeStore::new) { @Override public SailSource getExplicitSailSource() { diff --git a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/TripleStore.java b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/TripleStore.java index 3c060af663d..88e60a3eaa4 100644 --- a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/TripleStore.java +++ b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/TripleStore.java @@ -17,12 +17,16 @@ import java.io.IOException; import java.io.InputStream; import java.io.OutputStream; +import java.lang.invoke.MethodHandles; +import java.lang.invoke.VarHandle; +import java.nio.ByteOrder; import java.util.ArrayList; import java.util.Arrays; import java.util.Collections; import java.util.HashMap; import java.util.HashSet; import java.util.List; +import java.util.Locale; import java.util.Map; import java.util.Properties; import java.util.Set; @@ -76,6 +80,12 @@ class TripleStore implements Closeable { */ private static final String INDEXES_KEY = "triple-indexes"; + /** + * System property that enables the experimental {@link MemoryMappedTxnStatusFile} implementation instead of the + * default {@link TxnStatusFile}. + */ + private static final String MEMORY_MAPPED_TXN_STATUS_FILE_ENABLED_PROP = "org.eclipse.rdf4j.sail.nativerdf.MemoryMappedTxnStatusFile.enabled"; + /** * The version number for the current triple store. *

    @@ -164,7 +174,7 @@ public TripleStore(File dir, String indexSpecStr) throws IOException, SailExcept public TripleStore(File dir, String indexSpecStr, boolean forceSync) throws IOException, SailException { this.dir = dir; this.forceSync = forceSync; - this.txnStatusFile = new TxnStatusFile(dir); + this.txnStatusFile = createTxnStatusFile(dir); File propFile = new File(dir, PROPERTIES_FILE); @@ -219,6 +229,13 @@ public TripleStore(File dir, String indexSpecStr, boolean forceSync) throws IOEx } } + private static TxnStatusFile createTxnStatusFile(File dir) throws IOException { + if (Boolean.getBoolean(MEMORY_MAPPED_TXN_STATUS_FILE_ENABLED_PROP)) { + return new MemoryMappedTxnStatusFile(dir); + } + return new TxnStatusFile(dir); + } + /*---------* * Methods * *---------*/ @@ -936,7 +953,7 @@ private boolean shouldOverflowToDisk(RecordCache removedTriplesCache) { } public void startTransaction() throws IOException { - txnStatusFile.setTxnStatus(TxnStatus.ACTIVE); + txnStatusFile.setTxnStatus(TxnStatus.ACTIVE, forceSync); // Create a record cache for storing updated triples with a maximum of // some 10% of the number of triples @@ -951,7 +968,7 @@ public void startTransaction() throws IOException { } public void commit() throws IOException { - txnStatusFile.setTxnStatus(TxnStatus.COMMITTING); + txnStatusFile.setTxnStatus(TxnStatus.COMMITTING, forceSync); // updatedTriplesCache will be null when recovering from a crashed commit boolean validCache = updatedTriplesCache != null && updatedTriplesCache.isValid(); @@ -1006,7 +1023,7 @@ public void commit() throws IOException { sync(); - txnStatusFile.setTxnStatus(TxnStatus.NONE); + txnStatusFile.setTxnStatus(TxnStatus.NONE, forceSync); // checkAllCommitted(); } @@ -1029,7 +1046,7 @@ private void checkAllCommitted() throws IOException { } public void rollback() throws IOException { - txnStatusFile.setTxnStatus(TxnStatus.ROLLING_BACK); + txnStatusFile.setTxnStatus(TxnStatus.ROLLING_BACK, forceSync); // updatedTriplesCache will be null when recovering from a crash boolean validCache = updatedTriplesCache != null && updatedTriplesCache.isValid(); @@ -1083,7 +1100,7 @@ public void rollback() throws IOException { sync(); - txnStatusFile.setTxnStatus(TxnStatus.NONE); + txnStatusFile.setTxnStatus(TxnStatus.NONE, forceSync); } protected void sync() throws IOException { @@ -1196,7 +1213,8 @@ public TripleIndex(String fieldSeq, boolean deleteExistingIndexFile) throws IOEx } } tripleComparator = new TripleComparator(fieldSeq); - btree = new BTree(dir, getFilenamePrefix(fieldSeq), 2048, RECORD_LENGTH, tripleComparator, forceSync); + btree = new BTree(dir, getFilenamePrefix(fieldSeq), 2048, RECORD_LENGTH, tripleComparator.compareStrategy, + forceSync); } private String getFilenamePrefix(String fieldSeq) { @@ -1275,9 +1293,92 @@ public String toString() { private static class TripleComparator implements RecordComparator { private final char[] fieldSeq; + private final RecordComparator compareStrategy; public TripleComparator(String fieldSeq) { - this.fieldSeq = fieldSeq.toCharArray(); + String normalized = normalizeFieldSequence(fieldSeq); + this.fieldSeq = normalized.toCharArray(); + this.compareStrategy = getComparator(normalized); + } + + private static final RecordComparator compareSPOC = TripleComparator::compareSPOC; + private static final RecordComparator compareSPCO = TripleComparator::compareSPCO; + private static final RecordComparator compareSOPC = TripleComparator::compareSOPC; + private static final RecordComparator compareSOCP = TripleComparator::compareSOCP; + private static final RecordComparator compareSCPO = TripleComparator::compareSCPO; + private static final RecordComparator compareSCOP = TripleComparator::compareSCOP; + private static final RecordComparator comparePSOC = TripleComparator::comparePSOC; + private static final RecordComparator comparePSCO = TripleComparator::comparePSCO; + private static final RecordComparator comparePOSC = TripleComparator::comparePOSC; + private static final RecordComparator comparePOCS = TripleComparator::comparePOCS; + private static final RecordComparator comparePCSO = TripleComparator::comparePCSO; + private static final RecordComparator comparePCOS = TripleComparator::comparePCOS; + private static final RecordComparator compareOSPC = TripleComparator::compareOSPC; + private static final RecordComparator compareOSCP = TripleComparator::compareOSCP; + private static final RecordComparator compareOPSC = TripleComparator::compareOPSC; + private static final RecordComparator compareOPCS = TripleComparator::compareOPCS; + private static final RecordComparator compareOCSP = TripleComparator::compareOCSP; + private static final RecordComparator compareOCPS = TripleComparator::compareOCPS; + private static final RecordComparator compareCSPO = TripleComparator::compareCSPO; + private static final RecordComparator compareCSOP = TripleComparator::compareCSOP; + private static final RecordComparator compareCPSO = TripleComparator::compareCPSO; + private static final RecordComparator compareCPOS = TripleComparator::compareCPOS; + private static final RecordComparator compareCOSP = TripleComparator::compareCOSP; + private static final RecordComparator compareCOPS = TripleComparator::compareCOPS; + + private static RecordComparator getComparator(String order) { + switch (order) { + case "spoc": + return compareSPOC; + case "spco": + return compareSPCO; + case "sopc": + return compareSOPC; + case "socp": + return compareSOCP; + case "scpo": + return compareSCPO; + case "scop": + return compareSCOP; + case "psoc": + return comparePSOC; + case "psco": + return comparePSCO; + case "posc": + return comparePOSC; + case "pocs": + return comparePOCS; + case "pcso": + return comparePCSO; + case "pcos": + return comparePCOS; + case "ospc": + return compareOSPC; + case "oscp": + return compareOSCP; + case "opsc": + return compareOPSC; + case "opcs": + return compareOPCS; + case "ocsp": + return compareOCSP; + case "ocps": + return compareOCPS; + case "cspo": + return compareCSPO; + case "csop": + return compareCSOP; + case "cpso": + return compareCPSO; + case "cpos": + return compareCPOS; + case "cosp": + return compareCOSP; + case "cops": + return compareCOPS; + default: + throw new IllegalArgumentException("Unknown field order: " + order); + } } public char[] getFieldSeq() { @@ -1286,36 +1387,186 @@ public char[] getFieldSeq() { @Override public final int compareBTreeValues(byte[] key, byte[] data, int offset, int length) { - for (char field : fieldSeq) { - int fieldIdx; + return compareStrategy.compareBTreeValues(key, data, offset, length); + } - switch (field) { - case 's': - fieldIdx = SUBJ_IDX; - break; - case 'p': - fieldIdx = PRED_IDX; - break; - case 'o': - fieldIdx = OBJ_IDX; - break; - case 'c': - fieldIdx = CONTEXT_IDX; - break; - default: - throw new IllegalArgumentException( - "invalid character '" + field + "' in field sequence: " + new String(fieldSeq)); - } + private static String normalizeFieldSequence(String fieldSeq) { + if (fieldSeq == null) { + throw new IllegalArgumentException("Field sequence must not be null"); + } + String normalized = fieldSeq.trim().toLowerCase(Locale.ROOT); + if (normalized.length() != 4) { + throw new IllegalArgumentException( + "Field sequence '" + fieldSeq + "' must be four characters long (permutation of 'spoc')."); + } + return normalized; + } - int diff = ByteArrayUtil.compareRegion(key, fieldIdx, data, offset + fieldIdx, 4); + private static int compareSPOC(byte[] key, byte[] data, int offset, int length) { + return compareFields(key, data, offset, SUBJ_IDX, PRED_IDX, OBJ_IDX, CONTEXT_IDX); + } - if (diff != 0) { - return diff; - } - } + private static int compareSPCO(byte[] key, byte[] data, int offset, int length) { + return compareFields(key, data, offset, SUBJ_IDX, PRED_IDX, CONTEXT_IDX, OBJ_IDX); + } + + private static int compareSOPC(byte[] key, byte[] data, int offset, int length) { + return compareFields(key, data, offset, SUBJ_IDX, OBJ_IDX, PRED_IDX, CONTEXT_IDX); + } + + private static int compareSOCP(byte[] key, byte[] data, int offset, int length) { + return compareFields(key, data, offset, SUBJ_IDX, OBJ_IDX, CONTEXT_IDX, PRED_IDX); + } + + private static int compareSCPO(byte[] key, byte[] data, int offset, int length) { + return compareFields(key, data, offset, SUBJ_IDX, CONTEXT_IDX, PRED_IDX, OBJ_IDX); + } + + private static int compareSCOP(byte[] key, byte[] data, int offset, int length) { + return compareFields(key, data, offset, SUBJ_IDX, CONTEXT_IDX, OBJ_IDX, PRED_IDX); + } + + private static int comparePSOC(byte[] key, byte[] data, int offset, int length) { + return compareFields(key, data, offset, PRED_IDX, SUBJ_IDX, OBJ_IDX, CONTEXT_IDX); + } + + private static int comparePSCO(byte[] key, byte[] data, int offset, int length) { + return compareFields(key, data, offset, PRED_IDX, SUBJ_IDX, CONTEXT_IDX, OBJ_IDX); + } + + private static int comparePOSC(byte[] key, byte[] data, int offset, int length) { + return compareFields(key, data, offset, PRED_IDX, OBJ_IDX, SUBJ_IDX, CONTEXT_IDX); + } + + private static int comparePOCS(byte[] key, byte[] data, int offset, int length) { + return compareFields(key, data, offset, PRED_IDX, OBJ_IDX, CONTEXT_IDX, SUBJ_IDX); + } + + private static int comparePCSO(byte[] key, byte[] data, int offset, int length) { + return compareFields(key, data, offset, PRED_IDX, CONTEXT_IDX, SUBJ_IDX, OBJ_IDX); + } + + private static int comparePCOS(byte[] key, byte[] data, int offset, int length) { + return compareFields(key, data, offset, PRED_IDX, CONTEXT_IDX, OBJ_IDX, SUBJ_IDX); + } + + private static int compareOSPC(byte[] key, byte[] data, int offset, int length) { + return compareFields(key, data, offset, OBJ_IDX, SUBJ_IDX, PRED_IDX, CONTEXT_IDX); + } + + private static int compareOSCP(byte[] key, byte[] data, int offset, int length) { + return compareFields(key, data, offset, OBJ_IDX, SUBJ_IDX, CONTEXT_IDX, PRED_IDX); + } + + private static int compareOPSC(byte[] key, byte[] data, int offset, int length) { + return compareFields(key, data, offset, OBJ_IDX, PRED_IDX, SUBJ_IDX, CONTEXT_IDX); + } + + private static int compareOPCS(byte[] key, byte[] data, int offset, int length) { + return compareFields(key, data, offset, OBJ_IDX, PRED_IDX, CONTEXT_IDX, SUBJ_IDX); + } + + private static int compareOCSP(byte[] key, byte[] data, int offset, int length) { + return compareFields(key, data, offset, OBJ_IDX, CONTEXT_IDX, SUBJ_IDX, PRED_IDX); + } + + private static int compareOCPS(byte[] key, byte[] data, int offset, int length) { + return compareFields(key, data, offset, OBJ_IDX, CONTEXT_IDX, PRED_IDX, SUBJ_IDX); + } + + private static int compareCSPO(byte[] key, byte[] data, int offset, int length) { + return compareFields(key, data, offset, CONTEXT_IDX, SUBJ_IDX, PRED_IDX, OBJ_IDX); + } + + private static int compareCSOP(byte[] key, byte[] data, int offset, int length) { + return compareFields(key, data, offset, CONTEXT_IDX, SUBJ_IDX, OBJ_IDX, PRED_IDX); + } + + private static int compareCPSO(byte[] key, byte[] data, int offset, int length) { + return compareFields(key, data, offset, CONTEXT_IDX, PRED_IDX, SUBJ_IDX, OBJ_IDX); + } + + private static int compareCPOS(byte[] key, byte[] data, int offset, int length) { + return compareFields(key, data, offset, CONTEXT_IDX, PRED_IDX, OBJ_IDX, SUBJ_IDX); + } + + private static int compareCOSP(byte[] key, byte[] data, int offset, int length) { + return compareFields(key, data, offset, CONTEXT_IDX, OBJ_IDX, SUBJ_IDX, PRED_IDX); + } + + private static int compareCOPS(byte[] key, byte[] data, int offset, int length) { + return compareFields(key, data, offset, CONTEXT_IDX, OBJ_IDX, PRED_IDX, SUBJ_IDX); + } + + /** + * Lexicographically compares four 4-byte fields drawn from 'key' and 'data' at indices (first, second, third, + * fourth), where the data side is offset by 'offset'. Bytes are treated as unsigned, and the return value is + * the (unsigned) difference of the first mismatching bytes, or 0 if all four fields are equal. + */ + static int compareFields(byte[] key, byte[] data, int offset, + int first, int second, int third, int fourth) { + + // Field 1 + int a = (int) INT_BE.get(key, first); + int b = (int) INT_BE.get(data, offset + first); + int x = a ^ b; + if (x != 0) + return diffFromXorInt(a, b, x); + + // Field 2 + a = (int) INT_BE.get(key, second); + b = (int) INT_BE.get(data, offset + second); + x = a ^ b; + if (x != 0) + return diffFromXorInt(a, b, x); + + // Field 3 + a = (int) INT_BE.get(key, third); + b = (int) INT_BE.get(data, offset + third); + x = a ^ b; + if (x != 0) + return diffFromXorInt(a, b, x); + + // Field 4 + a = (int) INT_BE.get(key, fourth); + b = (int) INT_BE.get(data, offset + fourth); + x = a ^ b; + if (x != 0) + return diffFromXorInt(a, b, x); return 0; } + + /** + * Given two big-endian-packed ints and their XOR (non-zero), return the (unsigned) difference of the first + * mismatching bytes. + * + * Trick: the first differing byte’s position is the number of leading zeros of x, rounded down to a multiple of + * 8. Left-shift both ints by that many bits so the mismatching byte moves into the top byte, then extract it. + */ + private static int diffFromXorInt(int a, int b, int x) { + int n = Integer.numberOfLeadingZeros(x) & ~7; // 0,8,16,24 + return ((a << n) >>> 24) - ((b << n) >>> 24); + } + + private static final VarHandle INT_BE = MethodHandles.byteArrayViewVarHandle(int[].class, ByteOrder.BIG_ENDIAN); + + public static int compareFieldLength4(byte[] key, byte[] data, int offset, int fieldIdx) { + final int a = (int) INT_BE.get(key, fieldIdx); + final int b = (int) INT_BE.get(data, offset + fieldIdx); + + final int x = a ^ b; // mask of differing bits + if (x == 0) + return 0; // all 4 bytes equal + + // Find the first differing *byte* from the left (k .. k+3). + // With a big‑endian view, the first byte lives in bits 31..24, etc. + final int byteIndex = Integer.numberOfLeadingZeros(x) >>> 3; // 0..3 equal-leading-byte count + final int shift = 24 - (byteIndex << 3); + + // Extract that byte from each int (as unsigned) and return their difference. + return ((a >>> shift) & 0xFF) - ((b >>> shift) & 0xFF); + } } private static boolean isAssertionsEnabled() { diff --git a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/TxnStatusFile.java b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/TxnStatusFile.java index 3f7e85f22f9..311af43bdb7 100644 --- a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/TxnStatusFile.java +++ b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/TxnStatusFile.java @@ -68,13 +68,13 @@ byte[] getOnDisk() { return onDisk; } - private static final byte NONE_BYTE = (byte) 0b00000000; - private static final byte OLD_NONE_BYTE = (byte) 0b00000001; + static final byte NONE_BYTE = (byte) 0b00000000; + static final byte OLD_NONE_BYTE = (byte) 0b00000001; - private static final byte ACTIVE_BYTE = (byte) 0b00000010; - private static final byte COMMITTING_BYTE = (byte) 0b00000100; - private static final byte ROLLING_BACK_BYTE = (byte) 0b00001000; - private static final byte UNKNOWN_BYTE = (byte) 0b00010000; + static final byte ACTIVE_BYTE = (byte) 0b00000010; + static final byte COMMITTING_BYTE = (byte) 0b00000100; + static final byte ROLLING_BACK_BYTE = (byte) 0b00001000; + static final byte UNKNOWN_BYTE = (byte) 0b00010000; } @@ -96,8 +96,14 @@ public TxnStatusFile(File dataDir) throws IOException { nioFile = new NioFile(statusFile, "rwd"); } + public TxnStatusFile() { + nioFile = null; + } + public void close() throws IOException { - nioFile.close(); + if (nioFile != null) { + nioFile.close(); + } } /** @@ -106,15 +112,21 @@ public void close() throws IOException { * @param txnStatus The transaction status to write. * @throws IOException If the transaction status could not be written to file. */ - public void setTxnStatus(TxnStatus txnStatus) throws IOException { + public void setTxnStatus(TxnStatus txnStatus, boolean force) throws IOException { if (disabled) { return; } if (txnStatus == TxnStatus.NONE) { + // noinspection DataFlowIssue nioFile.truncate(0); } else { + // noinspection DataFlowIssue nioFile.writeBytes(txnStatus.onDisk, 0); } + + if (force) { + nioFile.force(false); + } } /** @@ -128,41 +140,28 @@ public TxnStatus getTxnStatus() throws IOException { if (disabled) { return TxnStatus.NONE; } - byte[] bytes; try { - bytes = nioFile.readBytes(0, 1); + // noinspection DataFlowIssue + return statusMapping[nioFile.readBytes(0, 1)[0]]; } catch (EOFException e) { // empty file = NONE status return TxnStatus.NONE; + } catch (IndexOutOfBoundsException e) { + // fall back to deprecated reading method + return getTxnStatusDeprecated(); } - TxnStatus status; - - switch (bytes[0]) { - case TxnStatus.NONE_BYTE: - status = TxnStatus.NONE; - break; - case TxnStatus.OLD_NONE_BYTE: - status = TxnStatus.NONE; - break; - case TxnStatus.ACTIVE_BYTE: - status = TxnStatus.ACTIVE; - break; - case TxnStatus.COMMITTING_BYTE: - status = TxnStatus.COMMITTING; - break; - case TxnStatus.ROLLING_BACK_BYTE: - status = TxnStatus.ROLLING_BACK; - break; - case TxnStatus.UNKNOWN_BYTE: - status = TxnStatus.UNKNOWN; - break; - default: - status = getTxnStatusDeprecated(); - } + } - return status; + final static TxnStatus[] statusMapping = new TxnStatus[17]; + static { + statusMapping[TxnStatus.NONE_BYTE] = TxnStatus.NONE; + statusMapping[TxnStatus.OLD_NONE_BYTE] = TxnStatus.NONE; + statusMapping[TxnStatus.ACTIVE_BYTE] = TxnStatus.ACTIVE; + statusMapping[TxnStatus.COMMITTING_BYTE] = TxnStatus.COMMITTING; + statusMapping[TxnStatus.ROLLING_BACK_BYTE] = TxnStatus.ROLLING_BACK; + statusMapping[TxnStatus.UNKNOWN_BYTE] = TxnStatus.UNKNOWN; } private TxnStatus getTxnStatusDeprecated() throws IOException { @@ -170,6 +169,7 @@ private TxnStatus getTxnStatusDeprecated() throws IOException { return TxnStatus.NONE; } + // noinspection DataFlowIssue byte[] bytes = nioFile.readBytes(0, (int) nioFile.size()); String s = new String(bytes, US_ASCII); diff --git a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/ValueStore.java b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/ValueStore.java index 9c4786bf27e..0fb53206afc 100644 --- a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/ValueStore.java +++ b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/ValueStore.java @@ -15,9 +15,21 @@ import java.io.File; import java.io.IOException; import java.io.UnsupportedEncodingException; +import java.net.URI; import java.nio.charset.StandardCharsets; +import java.util.ArrayList; import java.util.Arrays; +import java.util.Comparator; +import java.util.List; +import java.util.Map; +import java.util.Objects; import java.util.Optional; +import java.util.OptionalLong; +import java.util.concurrent.CancellationException; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.CompletionException; +import java.util.stream.Collectors; +import java.util.zip.CRC32C; import org.eclipse.rdf4j.common.annotation.InternalUseOnly; import org.eclipse.rdf4j.common.concurrent.locks.Lock; @@ -47,6 +59,13 @@ import org.eclipse.rdf4j.sail.nativerdf.model.NativeLiteral; import org.eclipse.rdf4j.sail.nativerdf.model.NativeResource; import org.eclipse.rdf4j.sail.nativerdf.model.NativeValue; +import org.eclipse.rdf4j.sail.nativerdf.wal.ValueStoreWAL; +import org.eclipse.rdf4j.sail.nativerdf.wal.ValueStoreWalConfig; +import org.eclipse.rdf4j.sail.nativerdf.wal.ValueStoreWalReader; +import org.eclipse.rdf4j.sail.nativerdf.wal.ValueStoreWalRecord; +import org.eclipse.rdf4j.sail.nativerdf.wal.ValueStoreWalRecovery; +import org.eclipse.rdf4j.sail.nativerdf.wal.ValueStoreWalSearch; +import org.eclipse.rdf4j.sail.nativerdf.wal.ValueStoreWalValueKind; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -58,10 +77,13 @@ * one release to the next. */ @InternalUseOnly -public class ValueStore extends SimpleValueFactory { +public class ValueStore extends SimpleValueFactory implements AutoCloseable { private static final Logger logger = LoggerFactory.getLogger(ValueStore.class); + private static final String WAL_RECOVERY_LOG_PROP = "org.eclipse.rdf4j.sail.nativerdf.valuestorewal.recoveryLog"; + private static final String WAL_RECOVERY_LOG = System.getProperty(WAL_RECOVERY_LOG_PROP, "debug").toLowerCase(); + /** * The default value cache size. */ @@ -97,7 +119,12 @@ public class ValueStore extends SimpleValueFactory { /** * Used to do the actual storage of values, once they're translated to byte arrays. */ + private final File dataDir; private final DataStore dataStore; + private final ValueStoreWAL wal; + private final ThreadLocal walPendingLsn; + private volatile CompletableFuture walBootstrapFuture; + private volatile ValueStoreWalSearch walSearch; /** * Lock manager used to prevent the removal of values over multiple method calls. Note that values can still be @@ -146,7 +173,13 @@ public ValueStore(File dataDir, boolean forceSync) throws IOException { public ValueStore(File dataDir, boolean forceSync, int valueCacheSize, int valueIDCacheSize, int namespaceCacheSize, int namespaceIDCacheSize) throws IOException { + this(dataDir, forceSync, valueCacheSize, valueIDCacheSize, namespaceCacheSize, namespaceIDCacheSize, null); + } + + public ValueStore(File dataDir, boolean forceSync, int valueCacheSize, int valueIDCacheSize, int namespaceCacheSize, + int namespaceIDCacheSize, ValueStoreWAL wal) throws IOException { super(); + this.dataDir = dataDir; dataStore = new DataStore(dataDir, FILENAME_PREFIX, forceSync, this); valueCache = new ConcurrentCache<>(valueCacheSize); @@ -154,7 +187,13 @@ public ValueStore(File dataDir, boolean forceSync, int valueCacheSize, int value namespaceCache = new ConcurrentCache<>(namespaceCacheSize); namespaceIDCache = new ConcurrentCache<>(namespaceIDCacheSize); + this.wal = wal; + this.walPendingLsn = wal != null ? ThreadLocal.withInitial(() -> ValueStoreWAL.NO_LSN) : null; + + autoRecoverValueStoreIfConfigured(); + setNewRevision(); + maybeScheduleWalBootstrap(); } @@ -196,33 +235,67 @@ public NativeValue getValue(int id) throws IOException { NativeValue resultValue = valueCache.get(cacheID); if (resultValue == null) { + boolean recoveredDirectlyFromWal = false; try { // Value not in cache, fetch it from file byte[] data = dataStore.getData(id); + if (data != null) { resultValue = data2value(id, data); - if (!(resultValue instanceof CorruptValue)) { - // Store value in cache - valueCache.put(cacheID, resultValue); + if (resultValue instanceof CorruptValue) { + NativeValue recovered = ((CorruptValue) resultValue).getRecovered(); + if (recovered != null) { + resultValue = recovered; + } + } else if (shouldValidateAgainstWal()) { + NativeValue walValue = recoverValueFromWal(id, false); + if (walValue != null && !valuesMatch(resultValue, walValue)) { + resultValue = walValue; + recoveredDirectlyFromWal = true; + } } + } else { + resultValue = recoverValueFromWal(id, false); + recoveredDirectlyFromWal = resultValue != null; } + } catch (RecoveredDataException rde) { byte[] recovered = rde.getData(); + CorruptValue corruptValue; if (recovered != null && recovered.length > 0) { byte t = recovered[0]; if (t == URI_VALUE) { - resultValue = new CorruptIRI(revision, id, null, recovered); + corruptValue = new CorruptIRI(revision, id, null, recovered); } else if (t == BNODE_VALUE) { - resultValue = new CorruptIRIOrBNode(revision, id, recovered); + corruptValue = new CorruptIRIOrBNode(revision, id, recovered); } else if (t == LITERAL_VALUE) { - resultValue = new CorruptLiteral(revision, id, recovered); + corruptValue = new CorruptLiteral(revision, id, recovered); } else { - resultValue = new CorruptUnknownValue(revision, id, recovered); + corruptValue = new CorruptUnknownValue(revision, id, recovered); } } else { - resultValue = new CorruptUnknownValue(revision, id, recovered); + corruptValue = new CorruptUnknownValue(revision, id, recovered); + } + + tryRecoverFromWal(id, corruptValue); + NativeValue recoveredValue = corruptValue.getRecovered(); + if (recoveredValue != null) { + resultValue = recoveredValue; + recoveredDirectlyFromWal = true; + } else { + resultValue = corruptValue; } } + + if (recoveredDirectlyFromWal && resultValue != null) { + logRecovered(id, resultValue); + logWalRepairHint(id); + } + + if (resultValue != null && !(resultValue instanceof CorruptValue)) { + // Store value in cache + valueCache.put(cacheID, resultValue); + } } return resultValue; @@ -380,22 +453,6 @@ private static String threadName() { return Thread.currentThread().getName(); } - private static String describeValue(Value value) { - if (value == null) { - return "null"; - } - String lexical; - try { - lexical = value.stringValue(); - } catch (Exception e) { - lexical = String.valueOf(value); - } - if (lexical.length() > 120) { - lexical = lexical.substring(0, 117) + "..."; - } - return value.getClass().getSimpleName() + '[' + lexical + ']'; - } - /** * Stores the supplied value and returns the ID that has been assigned to it. In case the value was already present, * the value will not be stored again and the ID of the existing value is returned. @@ -451,6 +508,7 @@ public synchronized int storeValue(Value value) throws IOException { // store which will handle duplicates byte[] valueData = value2data(value, true); + int previousMaxID = walEnabled() ? dataStore.getMaxID() : 0; if (valueData == null) { if (logger.isDebugEnabled()) { logger.debug("storeValue computed no data for value={} thread={}", describeValue(value), threadName()); @@ -468,6 +526,10 @@ public synchronized int storeValue(Value value) throws IOException { // Update cache valueIDCache.put(nv, id); + if (walEnabled() && id > previousMaxID) { + logMintedValue(id, nv); + } + if (logger.isDebugEnabled()) { logger.debug("storeValue stored value={} assigned id={} thread={} dataSummary={}", describeValue(nv), id, threadName(), summarize(valueData)); @@ -485,6 +547,18 @@ public void clear() throws IOException { try { Lock writeLock = lockManager.getWriteLock(); try { + + // Purge any existing WAL segments so a subsequent WAL recovery cannot + // resurrect values that were present before the clear(). + if (walEnabled()) { + try { + wal.purgeAllSegments(); + } catch (IOException e) { + logger.warn("Failed to purge ValueStore WAL during clear for {}", dataDir, e); + throw e; + } + } + dataStore.clear(); valueCache.clear(); @@ -515,7 +589,19 @@ public void sync() throws IOException { * * @throws IOException If an I/O error occurred. */ + @Override public void close() throws IOException { + CompletableFuture bootstrap = walBootstrapFuture; + if (bootstrap != null) { + try { + bootstrap.join(); + } catch (CompletionException e) { + Throwable cause = e.getCause() == null ? e : e.getCause(); + logger.warn("ValueStore WAL bootstrap failed during close", cause); + } catch (CancellationException e) { + logger.warn("ValueStore WAL bootstrap was cancelled during close"); + } + } dataStore.close(); } @@ -537,7 +623,7 @@ public void checkConsistency() throws SailException, IOException { String namespace = data2namespace(data); try { if (id == getNamespaceID(namespace, false) - && java.net.URI.create(namespace + "part").isAbsolute()) { + && URI.create(namespace + "part").isAbsolute()) { continue; } } catch (IllegalArgumentException e) { @@ -729,7 +815,9 @@ public NativeValue data2value(int id, byte[] data) throws IOException { if (data.length == 0) { if (SOFT_FAIL_ON_CORRUPT_DATA_AND_REPAIR_INDEXES) { logger.error("Soft fail on corrupt data: Empty data array for value with id {}", id); - return new CorruptUnknownValue(revision, id, data); + CorruptUnknownValue v = new CorruptUnknownValue(revision, id, data); + tryRecoverFromWal(id, v); + return v; } throw new SailException("Empty data array for value with id " + id + " consider setting the system property org.eclipse.rdf4j.sail.nativerdf.softFailOnCorruptDataAndRepairIndexes to true"); @@ -744,7 +832,9 @@ public NativeValue data2value(int id, byte[] data) throws IOException { default: if (SOFT_FAIL_ON_CORRUPT_DATA_AND_REPAIR_INDEXES) { logger.error("Soft fail on corrupt data: Invalid type {} for value with id {}", data[0], id); - return new CorruptUnknownValue(revision, id, data); + CorruptUnknownValue v = new CorruptUnknownValue(revision, id, data); + tryRecoverFromWal(id, v); + return v; } throw new SailException("Invalid type " + data[0] + " for value with id " + id + " consider setting the system property org.eclipse.rdf4j.sail.nativerdf.softFailOnCorruptDataAndRepairIndexes to true"); @@ -764,13 +854,14 @@ private T data2uri(int id, byte[] data) throws IOE } catch (Throwable e) { if (SOFT_FAIL_ON_CORRUPT_DATA_AND_REPAIR_INDEXES && (e instanceof Exception || e instanceof AssertionError)) { - return (T) new CorruptIRI(revision, id, namespace, data); + CorruptIRI v = new CorruptIRI(revision, id, namespace, data); + tryRecoverFromWal(id, v); + return (T) v; } logger.warn( "NativeStore is possibly corrupt. To attempt to repair or retrieve the data, read the documentation on http://rdf4j.org about the system property org.eclipse.rdf4j.sail.nativerdf.softFailOnCorruptDataAndRepairIndexes"); throw e; } - } private NativeBNode data2bnode(int id, byte[] data) { @@ -807,13 +898,148 @@ private T data2literal(int id, byte[] data) th } catch (Throwable e) { if (SOFT_FAIL_ON_CORRUPT_DATA_AND_REPAIR_INDEXES && (e instanceof Exception || e instanceof AssertionError)) { - return (T) new CorruptLiteral(revision, id, data); + CorruptLiteral v = new CorruptLiteral(revision, id, data); + tryRecoverFromWal(id, v); + return (T) v; } throw e; } } + private void tryRecoverFromWal(int id, CorruptValue holder) { + NativeValue recovered = recoverValueFromWal(id); + if (recovered != null) { + holder.setRecovered(recovered); + } + } + + private NativeValue recoverValueFromWal(int id) { + return recoverValueFromWal(id, true); + } + + private NativeValue recoverValueFromWal(int id, boolean log) { + ValueStoreWalSearch search = getOrCreateWalSearch(); + if (search == null) { + return null; + } + try { + Value v = search.findValueById(id); + if (v == null) { + return null; + } + NativeValue nv = getNativeValue(v); + if (nv != null) { + nv.setInternalID(id, revision); + if (log) { + logRecovered(id, nv); + logWalRepairHint(id); + } + return nv; + } + } catch (IOException ioe) { + // ignore recovery failures + } + return null; + } + + private ValueStoreWalSearch getOrCreateWalSearch() { + if (wal == null) { + return null; + } + ValueStoreWalSearch search = walSearch; + if (search != null) { + return search; + } + synchronized (this) { + search = walSearch; + if (search == null) { + search = ValueStoreWalSearch.open(wal.config()); + walSearch = search; + } + return search; + } + } + + private boolean shouldValidateAgainstWal() { + return walEnabled() && SOFT_FAIL_ON_CORRUPT_DATA_AND_REPAIR_INDEXES; + } + + private boolean valuesMatch(NativeValue storeValue, NativeValue walValue) { + if (storeValue == walValue) { + return true; + } + if (storeValue == null || walValue == null) { + return false; + } + if (storeValue instanceof Literal && walValue instanceof Literal) { + Literal a = (Literal) storeValue; + Literal b = (Literal) walValue; + return Objects.equals(a.getLabel(), b.getLabel()) + && Objects.equals(a.getLanguage().orElse(null), b.getLanguage().orElse(null)) + && Objects.equals(datatypeIri(a), datatypeIri(b)); + } + if (storeValue instanceof IRI && walValue instanceof IRI) { + return Objects.equals(storeValue.stringValue(), walValue.stringValue()); + } + if (storeValue instanceof BNode && walValue instanceof BNode) { + return Objects.equals(storeValue.stringValue(), walValue.stringValue()); + } + return Objects.equals(storeValue.stringValue(), walValue.stringValue()); + } + + private String datatypeIri(Literal literal) { + return literal.getDatatype() == null ? "" : literal.getDatatype().stringValue(); + } + + private void logRecovered(int id, NativeValue nv) { + switch (WAL_RECOVERY_LOG) { + case "trace": + if (logger.isTraceEnabled()) { + logger.trace("Recovered value for id {} from WAL as {}", id, nv.stringValue()); + } + break; + case "debug": + if (logger.isDebugEnabled()) { + logger.debug("Recovered value for id {} from WAL as {}", id, nv.stringValue()); + } + break; + default: + // off or unknown: no-op + } + } + + private void logWalRepairHint(int id) { + logger.error( + "ValueStore {} recovered value id {} from WAL because the values.* files are corrupt. Enable NativeStore#setWalAutoRecoverOnOpen(true) (config:native.walAutoRecoverOnOpen) and restart, or run ValueStoreWalRecovery to replay the WAL and rebuild values.dat/values.id/values.hash so the on-disk data matches the WAL again.", + dataDir, id); + } + + private NativeValue fromWalRecord(ValueStoreWalRecord rec) { + switch (rec.valueKind()) { + case IRI: + return createIRI(rec.lexical()); + case BNODE: + return createBNode(rec.lexical()); + case LITERAL: { + String lang = rec.language(); + String dt = rec.datatype(); + if (lang != null && !lang.isEmpty()) { + return createLiteral(rec.lexical(), lang); + } else if (dt != null && !dt.isEmpty()) { + return createLiteral(rec.lexical(), createIRI(dt)); + } else { + return createLiteral(rec.lexical()); + } + } + case NAMESPACE: + // not a value; nothing to recover + return null; + default: + return null; + } + } + private String data2namespace(byte[] data) { return new String(data, StandardCharsets.UTF_8); } @@ -835,7 +1061,11 @@ private int getNamespaceID(String namespace, boolean create) throws IOException int id; if (create) { + int previousMaxID = walEnabled() ? dataStore.getMaxID() : 0; id = dataStore.storeData(namespaceData); + if (walEnabled() && id > previousMaxID) { + logNamespaceMint(id, namespace); + } } else { id = dataStore.getID(namespaceData); } @@ -852,6 +1082,218 @@ private int getNamespaceID(String namespace, boolean create) throws IOException return id; } + public OptionalLong drainPendingWalHighWaterMark() { + if (walPendingLsn == null) { + return OptionalLong.empty(); + } + long lsn = walPendingLsn.get(); + if (lsn <= ValueStoreWAL.NO_LSN) { + return OptionalLong.empty(); + } + walPendingLsn.set(ValueStoreWAL.NO_LSN); + return OptionalLong.of(lsn); + } + + public void awaitWalDurable(long lsn) throws IOException { + if (!walEnabled() || lsn <= ValueStoreWAL.NO_LSN) { + return; + } + try { + wal.awaitDurable(lsn); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new IOException("Interrupted while awaiting WAL durability", e); + } + } + + private void logMintedValue(int id, Value value) throws IOException { + ValueStoreWalDescription description = describeValue(value); + int hash = computeWalHash(description.kind, description.lexical, description.datatype, description.language); + long lsn = wal.logMint(id, description.kind, description.lexical, description.datatype, description.language, + hash); + recordWalLsn(lsn); + } + + private void logNamespaceMint(int id, String namespace) throws IOException { + int hash = computeWalHash(ValueStoreWalValueKind.NAMESPACE, namespace, "", ""); + long lsn = wal.logMint(id, ValueStoreWalValueKind.NAMESPACE, namespace, "", "", hash); + recordWalLsn(lsn); + } + + private void maybeScheduleWalBootstrap() { + if (!walEnabled()) { + return; + } + int maxId = dataStore.getMaxID(); + if (maxId <= 0) { + return; + } + boolean needsBootstrap = !wal.hasInitialSegments() || walNeedsBootstrap(maxId); + if (!needsBootstrap) { + return; + } + boolean syncBootstrap = false; + try { + syncBootstrap = wal.config().syncBootstrapOnOpen(); + } catch (Throwable ignore) { + // defensive: if config not accessible, default to async + } + if (syncBootstrap) { + // Perform bootstrap synchronously before allowing any further operations + rebuildWalFromExistingValues(maxId); + } else { + if (walBootstrapFuture != null) { + return; + } + CompletableFuture future = CompletableFuture.runAsync(() -> rebuildWalFromExistingValues(maxId)); + walBootstrapFuture = future; + future.whenComplete((unused, throwable) -> { + if (throwable != null) { + logger.warn("ValueStore WAL bootstrap failed", throwable); + } + }); + } + } + + private void rebuildWalFromExistingValues(int maxId) { + try { + for (int id = 1; id <= maxId; id++) { + if (Thread.currentThread().isInterrupted()) { + Thread.currentThread().interrupt(); + return; + } + if (wal.isClosed()) { + return; + } + byte[] data; + try { + data = dataStore.getData(id); + } catch (IOException e) { + logger.warn("Failed to read value {} while rebuilding WAL", id, e); + continue; + } + if (data == null) { + continue; + } + try { + if (isNamespaceData(data)) { + String namespace = data2namespace(data); + logNamespaceMint(id, namespace); + } else { + NativeValue value = data2value(id, data); + if (value != null) { + logMintedValue(id, value); + } + } + } catch (IOException e) { + if (wal.isClosed()) { + return; + } + logger.warn("Failed to rebuild WAL entry for id {}", id, e); + } catch (RuntimeException e) { + logger.warn("Unexpected failure while rebuilding WAL entry for id {}", id, e); + } + } + if (!wal.isClosed()) { + OptionalLong pending = drainPendingWalHighWaterMark(); + if (pending.isPresent()) { + awaitWalDurable(pending.getAsLong()); + } + } + } catch (Throwable t) { + logger.warn("Error while rebuilding ValueStore WAL", t); + } + } + + private boolean walNeedsBootstrap(int maxId) { + try (ValueStoreWalReader reader = ValueStoreWalReader.open(wal.config())) { + ValueStoreWalRecovery recovery = new ValueStoreWalRecovery(); + ValueStoreWalRecovery.ReplayReport report = recovery.replayWithReport(reader); + Map dict = report.dictionary(); + if (dict.isEmpty()) { + return true; + } + if (!report.complete()) { + return true; + } + for (int id = 1; id <= maxId; id++) { + if (!dict.containsKey(id)) { + return true; + } + } + return false; + } catch (IOException e) { + // if we cannot inspect WAL, avoid scheduling to not interfere with normal operations + return false; + } + } + + private void recordWalLsn(long lsn) { + if (walPendingLsn == null || lsn <= ValueStoreWAL.NO_LSN) { + return; + } + long current = walPendingLsn.get(); + if (lsn > current) { + walPendingLsn.set(lsn); + } + } + + private ValueStoreWalDescription describeValue(Value value) { + if (value instanceof IRI) { + return new ValueStoreWalDescription(ValueStoreWalValueKind.IRI, value.stringValue(), "", ""); + } else if (value instanceof BNode) { + return new ValueStoreWalDescription(ValueStoreWalValueKind.BNODE, value.stringValue(), "", ""); + } else if (value instanceof Literal) { + Literal literal = (Literal) value; + String lang = literal.getLanguage().orElse(""); + String datatype = literal.getDatatype() != null ? literal.getDatatype().stringValue() : ""; + return new ValueStoreWalDescription(ValueStoreWalValueKind.LITERAL, literal.getLabel(), datatype, lang); + } else { + throw new IllegalArgumentException("value parameter should be a URI, BNode or Literal"); + } + } + + private int computeWalHash(ValueStoreWalValueKind kind, String lexical, String datatype, String language) { + CRC32C crc32c = CRC32C_HOLDER.get(); + // Reset the checksum to ensure each computed hash reflects only the current value + crc32c.reset(); + crc32c.update((byte) kind.code()); + updateCrc(crc32c, lexical); + crc32c.update((byte) 0); + updateCrc(crc32c, datatype); + crc32c.update((byte) 0); + updateCrc(crc32c, language); + return (int) crc32c.getValue(); + } + + private void updateCrc(CRC32C crc32c, String value) { + if (value == null || value.isEmpty()) { + return; + } + byte[] bytes = value.getBytes(StandardCharsets.UTF_8); + crc32c.update(bytes, 0, bytes.length); + } + + private boolean walEnabled() { + return wal != null; + } + + private static final ThreadLocal CRC32C_HOLDER = ThreadLocal.withInitial(CRC32C::new); + + private static final class ValueStoreWalDescription { + final ValueStoreWalValueKind kind; + final String lexical; + final String datatype; + final String language; + + ValueStoreWalDescription(ValueStoreWalValueKind kind, String lexical, String datatype, String language) { + this.kind = kind; + this.lexical = lexical == null ? "" : lexical; + this.datatype = datatype == null ? "" : datatype; + this.language = language == null ? "" : language; + } + } + private String getNamespace(int id) throws IOException { Integer cacheID = id; String namespace = namespaceCache.get(cacheID); @@ -1001,4 +1443,200 @@ public static void main(String[] args) throws Exception { } } } + + private void autoRecoverValueStoreIfConfigured() { + if (wal == null) { + return; + } + ValueStoreWalConfig config; + try { + config = wal.config(); + } catch (Throwable t) { + logger.warn("ValueStore WAL configuration unavailable for {}", dataDir, t); + return; + } + if (!config.recoverValueStoreOnOpen()) { + return; + } + try { + ValueStoreWalRecovery recovery = new ValueStoreWalRecovery(); + ValueStoreWalRecovery.ReplayReport report; + try (ValueStoreWalReader reader = ValueStoreWalReader.open(config)) { + report = recovery.replayWithReport(reader); + } + Map dictionary = report.dictionary(); + if (dictionary.isEmpty()) { + return; + } + if (!report.complete()) { + logger.warn("Skipping ValueStore WAL recovery for {}: WAL segments incomplete", dataDir); + return; + } + if (hasDictionaryGaps(dictionary)) { + logger.warn("Skipping ValueStore WAL recovery for {}: WAL dictionary has gaps", dataDir); + return; + } + if (!shouldRecoverFromWalDictionary(dictionary)) { + return; + } + recoverValueStoreFromWal(dictionary); + logAutoRecovery(dictionary.size()); + } catch (IOException e) { + logger.warn("ValueStore WAL recovery failed for {}", dataDir, e); + } + } + + private boolean hasDictionaryGaps(Map dictionary) { + int maxId = dictionary.keySet().stream().mapToInt(Integer::intValue).max().orElse(0); + if (maxId <= 0) { + return false; + } + if (dictionary.size() == maxId) { + return false; + } + for (int expected = 1; expected <= maxId; expected++) { + if (!dictionary.containsKey(expected)) { + return true; + } + } + return false; + } + + private boolean shouldRecoverFromWalDictionary(Map dictionary) { + int maxWalId = dictionary.keySet().stream().mapToInt(Integer::intValue).max().orElse(0); + if (maxWalId <= 0) { + return false; + } + int currentMaxId = dataStore.getMaxID(); + if (currentMaxId == 0 && maxWalId > 0) { + return true; + } + if (currentMaxId < maxWalId) { + return true; + } + List ids = new ArrayList<>(dictionary.keySet()); + if (ids.isEmpty()) { + return false; + } + ids.sort(Integer::compareTo); + for (Integer id : ids) { + if (isMissingValueData(id)) { + return true; + } + } + return false; + } + + private boolean isMissingValueData(int id) { + if (id <= 0) { + return false; + } + try { + byte[] data = dataStore.getData(id); + return data == null || data.length == 0; + } catch (IOException e) { + return true; + } + } + + private void recoverValueStoreFromWal(Map dictionary) throws IOException { + dataStore.clear(); + valueCache.clear(); + valueIDCache.clear(); + namespaceCache.clear(); + namespaceIDCache.clear(); + + List> entries = dictionary.entrySet() + .stream() + .sorted(Map.Entry.comparingByKey(Comparator.naturalOrder())) + .collect(Collectors.toList()); + + for (Map.Entry entry : entries) { + ValueStoreWalRecord record = entry.getValue(); + byte[] data; + switch (record.valueKind()) { + case NAMESPACE: + data = record.lexical().getBytes(StandardCharsets.UTF_8); + break; + case IRI: + data = encodeIri(record.lexical(), dataStore); + break; + case BNODE: { + byte[] idBytes = record.lexical().getBytes(StandardCharsets.UTF_8); + data = new byte[1 + idBytes.length]; + data[0] = BNODE_VALUE; + ByteArrayUtil.put(idBytes, data, 1); + break; + } + case LITERAL: + data = encodeLiteral(record.lexical(), record.datatype(), record.language(), dataStore); + break; + default: + continue; + } + if (data == null) { + continue; + } + int assigned = dataStore.storeData(data); + if (assigned != record.id()) { + throw new IOException("ValueStore WAL recovery produced mismatched id " + assigned + + " (expected " + record.id() + ")"); + } + } + dataStore.sync(); + } + + private void logAutoRecovery(int recoveredCount) { + switch (WAL_RECOVERY_LOG) { + case "trace": + if (logger.isTraceEnabled()) { + logger.trace("Recovered {} ValueStore entries from WAL for {}", recoveredCount, dataDir); + } + break; + case "debug": + if (logger.isDebugEnabled()) { + logger.debug("Recovered {} ValueStore entries from WAL for {}", recoveredCount, dataDir); + } + break; + default: + // off + } + } + + private byte[] encodeIri(String lexical, DataStore ds) throws IOException { + IRI iri = createIRI(lexical); + String ns = iri.getNamespace(); + String local = iri.getLocalName(); + int nsId = ds.getID(ns.getBytes(StandardCharsets.UTF_8)); + if (nsId == -1) { + nsId = ds.storeData(ns.getBytes(StandardCharsets.UTF_8)); + } + byte[] localBytes = local.getBytes(StandardCharsets.UTF_8); + byte[] data = new byte[1 + 4 + localBytes.length]; + data[0] = URI_VALUE; + ByteArrayUtil.putInt(nsId, data, 1); + ByteArrayUtil.put(localBytes, data, 5); + return data; + } + + private byte[] encodeLiteral(String label, String datatype, String language, DataStore ds) throws IOException { + int dtId = NativeValue.UNKNOWN_ID; + if (datatype != null && !datatype.isEmpty()) { + byte[] dtBytes = encodeIri(datatype, ds); + int id = ds.getID(dtBytes); + dtId = id == -1 ? ds.storeData(dtBytes) : id; + } + byte[] langBytes = language == null ? new byte[0] : language.getBytes(StandardCharsets.UTF_8); + byte[] labelBytes = label.getBytes(StandardCharsets.UTF_8); + byte[] data = new byte[1 + 4 + 1 + langBytes.length + labelBytes.length]; + data[0] = LITERAL_VALUE; + ByteArrayUtil.putInt(dtId, data, 1); + data[5] = (byte) (langBytes.length & 0xFF); + if (langBytes.length > 0) { + ByteArrayUtil.put(langBytes, data, 6); + } + ByteArrayUtil.put(labelBytes, data, 6 + langBytes.length); + return data; + } + } diff --git a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/btree/AllocatedNodesList.java b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/btree/AllocatedNodesList.java index a092a278b59..d20b2438bd0 100644 --- a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/btree/AllocatedNodesList.java +++ b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/btree/AllocatedNodesList.java @@ -13,16 +13,21 @@ import java.io.Closeable; import java.io.File; import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.MappedByteBuffer; import java.nio.channels.FileChannel; +import java.nio.file.StandardOpenOption; import java.util.Arrays; import java.util.BitSet; import org.eclipse.rdf4j.common.io.ByteArrayUtil; -import org.eclipse.rdf4j.common.io.NioFile; /** * List of allocated BTree nodes, persisted to a file on disk. * + * Incremental mmap version: node allocations/frees update the on-disk bitfield in-place, without rewriting the full + * bitmap on every sync. + * * @author Arjohn Kampman */ class AllocatedNodesList implements Closeable { @@ -56,7 +61,23 @@ class AllocatedNodesList implements Closeable { /** * The allocated nodes file. */ - private final NioFile nioFile; + private final File allocNodesFile; + + /** + * File channel used for reading and writing the allocated nodes file. + */ + private final FileChannel channel; + + /** + * Memory-mapped buffer for the entire file: header + bitfield. + */ + private MappedByteBuffer mapped; + + /** + * Number of bits that can currently be represented by the on-disk bitfield. This is (mapped.capacity() - + * HEADER_LENGTH) * 8. + */ + private int bitCapacity = 0; /** * Bit set recording which nodes have been allocated, using node IDs as index. @@ -64,7 +85,7 @@ class AllocatedNodesList implements Closeable { private BitSet allocatedNodes; /** - * Flag indicating whether the set of allocated nodes has changed and needs to be written to file. + * Flag indicating whether the set of allocated nodes has changed and needs to be synced (force()). */ private boolean needsSync = false; @@ -88,9 +109,20 @@ public AllocatedNodesList(File allocNodesFile, BTree btree, boolean forceSync) t throw new IllegalArgumentException("btree muts not be null"); } - this.nioFile = new NioFile(allocNodesFile); + this.allocNodesFile = allocNodesFile; this.btree = btree; this.forceSync = forceSync; + + this.channel = FileChannel.open( + allocNodesFile.toPath(), + StandardOpenOption.READ, + StandardOpenOption.WRITE, + StandardOpenOption.CREATE); + + // We delay actual mapping until we know the desired bitset size + // (after initAllocatedNodes / loadAllocatedNodesInfo / crawlAllocatedNodes). + this.mapped = null; + this.bitCapacity = 64; } /*---------* @@ -101,7 +133,7 @@ public AllocatedNodesList(File allocNodesFile, BTree btree, boolean forceSync) t * Gets the allocated nodes file. */ public File getFile() { - return nioFile.getFile(); + return allocNodesFile; } @Override @@ -116,7 +148,7 @@ public synchronized void close() throws IOException { */ public synchronized boolean delete() throws IOException { close(false); - return nioFile.delete(); + return allocNodesFile.delete(); } public synchronized void close(boolean syncChanges) throws IOException { @@ -125,42 +157,30 @@ public synchronized void close(boolean syncChanges) throws IOException { } allocatedNodes = null; needsSync = false; - nioFile.close(); + mapped = null; // let GC clean up mapping + channel.close(); } /** * Writes any changes that are cached in memory to disk. * - * @throws IOException + * For mmap, changes to individual bits are already reflected in the mapped region; sync() is mainly responsible for + * calling force() when requested. */ public synchronized void sync() throws IOException { - if (needsSync) { - // Trim bit set - BitSet bitSet = allocatedNodes; - int bitSetLength = allocatedNodes.length(); - if (bitSetLength < allocatedNodes.size()) { - bitSet = allocatedNodes.get(0, bitSetLength); - } - - byte[] data = ByteArrayUtil.toByteArray(bitSet); - - // Write bit set to file - nioFile.truncate(HEADER_LENGTH + data.length); - nioFile.writeBytes(MAGIC_NUMBER, 0); - nioFile.writeByte(FILE_FORMAT_VERSION, MAGIC_NUMBER.length); - nioFile.writeBytes(data, HEADER_LENGTH); - - if (forceSync) { - nioFile.force(false); - } + if (!needsSync) { + return; + } - needsSync = false; + if (mapped != null && forceSync) { + mapped.force(); } + + needsSync = false; } - private void scheduleSync() throws IOException { - if (needsSync == false) { - nioFile.truncate(0); + private void scheduleSync() { + if (!needsSync) { needsSync = true; } } @@ -171,11 +191,18 @@ private void scheduleSync() throws IOException { * @throws IOException If an I/O error occurred. */ public synchronized void clear() throws IOException { - if (allocatedNodes != null) { - allocatedNodes.clear(); - } else { - // bit set has not yet been initialized - allocatedNodes = new BitSet(); + initAllocatedNodes(); + + allocatedNodes.clear(); + + // Clear on-disk bits as well (if mapped and any capacity). + if (mapped != null && bitCapacity > 0) { + int byteCount = (bitCapacity + 7) >>> 3; + int start = HEADER_LENGTH; + int end = start + byteCount; + for (int pos = start; pos < end; pos++) { + mapped.put(pos, (byte) 0); + } } scheduleSync(); @@ -187,6 +214,9 @@ public synchronized int allocateNode() throws IOException { int newNodeID = allocatedNodes.nextClearBit(1); allocatedNodes.set(newNodeID); + ensureCapacityForBit(newNodeID); + setOnDiskBit(newNodeID, true); + scheduleSync(); return newNodeID; @@ -194,7 +224,16 @@ public synchronized int allocateNode() throws IOException { public synchronized void freeNode(int nodeID) throws IOException { initAllocatedNodes(); + allocatedNodes.clear(nodeID); + + // It's possible we free a node above current bitCapacity if the file + // was truncated, but in normal operation ensureCapacityForBit() will + // have made sure we have space for this bit already. + if (bitCapacity > 0 && nodeID < bitCapacity && mapped != null) { + setOnDiskBit(nodeID, false); + } + scheduleSync(); } @@ -214,37 +253,84 @@ public synchronized int getNodeCount() throws IOException { return allocatedNodes.cardinality(); } + /*--------------* + * Initialization * + *--------------*/ + private void initAllocatedNodes() throws IOException { - if (allocatedNodes == null) { - if (nioFile.size() > 0L) { - loadAllocatedNodesInfo(); - } else { - crawlAllocatedNodes(); - } + if (allocatedNodes != null) { + return; } + + long size = channel.size(); + if (size > 0L) { + loadAllocatedNodesInfo(); + } else { + crawlAllocatedNodes(); + } + + // At this point allocatedNodes is initialized; we can build an mmap + // representing the current state so that future alloc/free calls + // can update bits incrementally. + remapFromAllocatedNodes(); } + /** + * Load allocated node info from disk (old or new format), into the in-memory BitSet. + */ private void loadAllocatedNodesInfo() throws IOException { + long size = channel.size(); + if (size <= 0L) { + allocatedNodes = new BitSet(); + return; + } + + // We read using standard I/O so we can interpret both headered and + // headerless (old) formats. + ByteBuffer buf = ByteBuffer.allocate((int) size); + channel.position(0L); + while (buf.hasRemaining()) { + if (channel.read(buf) < 0) { + break; + } + } + byte[] fileBytes = buf.array(); + byte[] data; - if (nioFile.size() >= HEADER_LENGTH && Arrays.equals(MAGIC_NUMBER, nioFile.readBytes(0, MAGIC_NUMBER.length))) { - byte version = nioFile.readByte(MAGIC_NUMBER.length); + if (size >= HEADER_LENGTH && hasMagicHeader(fileBytes)) { + byte version = fileBytes[MAGIC_NUMBER.length]; if (version > FILE_FORMAT_VERSION) { throw new IOException("Unable to read allocated nodes file; it uses a newer file format"); } else if (version != FILE_FORMAT_VERSION) { throw new IOException("Unable to read allocated nodes file; invalid file format version: " + version); } - data = nioFile.readBytes(HEADER_LENGTH, (int) (nioFile.size() - HEADER_LENGTH)); + int dataLength = (int) (size - HEADER_LENGTH); + data = new byte[dataLength]; + System.arraycopy(fileBytes, HEADER_LENGTH, data, 0, dataLength); } else { // assume header is missing (old file format) - data = nioFile.readBytes(0, (int) nioFile.size()); + data = fileBytes; + // triggers rewrite to new headered format on next sync scheduleSync(); } allocatedNodes = ByteArrayUtil.toBitSet(data); } + private boolean hasMagicHeader(byte[] fileBytes) { + if (fileBytes.length < MAGIC_NUMBER.length) { + return false; + } + for (int i = 0; i < MAGIC_NUMBER.length; i++) { + if (fileBytes[i] != MAGIC_NUMBER[i]) { + return false; + } + } + return true; + } + private void crawlAllocatedNodes() throws IOException { allocatedNodes = new BitSet(); @@ -253,6 +339,7 @@ private void crawlAllocatedNodes() throws IOException { crawlAllocatedNodes(rootNode); } + // after crawling, we will write a fresh header+bitmap scheduleSync(); } @@ -265,9 +352,131 @@ private void crawlAllocatedNodes(Node node) throws IOException { crawlAllocatedNodes(node.getChildNode(i)); } } - } finally { node.release(); } } + + /*--------------* + * mmap helpers * + *--------------*/ + + /** + * Ensure that the mapped file has enough room to represent the given bit index. If not, grow the file and rebuild + * the mapping from the current BitSet. + */ + private void ensureCapacityForBit(int bitIndex) throws IOException { + // bits start at index 0; we need space for [0..bitIndex] + int neededBits = bitIndex + 1; + if (neededBits <= bitCapacity && mapped != null) { + return; + } + + // Expand capacity to at least neededBits, rounded up to a multiple of 64 bits + int newBitCapacity = Math.max(neededBits, bitCapacity); + newBitCapacity = (newBitCapacity + (4 * 8 * 1024) - 1) & ~((4 * 8 * 1024) - 1); // round up to 4KB boundary + newBitCapacity -= HEADER_LENGTH * 8; + + assert newBitCapacity > 0; + if (newBitCapacity < 0) { + newBitCapacity = neededBits + 8; // at least 8 bits + } + + // Serialize current BitSet into bytes according to the existing format + byte[] data = ByteArrayUtil.toByteArray(allocatedNodes); + int neededBytes = (newBitCapacity + 7) >>> 3; + if (data.length < neededBytes) { + data = Arrays.copyOf(data, neededBytes); + } + + long newFileSize = HEADER_LENGTH + (long) data.length; + + // Resize file on disk + long currentSize = channel.size(); + if (currentSize < newFileSize) { + channel.position(newFileSize - 1); + channel.write(ByteBuffer.wrap(new byte[] { 0 })); + } else if (currentSize > newFileSize) { + channel.truncate(newFileSize); + } + + // Remap and write header + data + mapped = channel.map(FileChannel.MapMode.READ_WRITE, 0, newFileSize); + mapped.position(0); + mapped.put(MAGIC_NUMBER); + mapped.put(FILE_FORMAT_VERSION); + mapped.put(data); + + bitCapacity = newBitCapacity; + } + + /** + * Rebuild the mmap and on-disk representation from the current in-memory BitSet. Used at initialization / migration + * time. + */ + private void remapFromAllocatedNodes() throws IOException { + // Determine minimal bit capacity needed for current BitSet + int neededBits = Math.max(allocatedNodes.length(), 1); // at least 1 bit + int newBitCapacity = (neededBits + (4 * 8 * 1024) - 1) & ~((4 * 8 * 1024) - 1); // round up to 4KB boundary + newBitCapacity -= HEADER_LENGTH * 8; + + assert newBitCapacity > 0; + if (newBitCapacity < 0) { + newBitCapacity = neededBits + 8; // at least 8 bits + } + + byte[] data = ByteArrayUtil.toByteArray(allocatedNodes); + int neededBytes = (newBitCapacity + 7) >>> 3; + if (data.length < neededBytes) { + data = Arrays.copyOf(data, neededBytes); + } + + long newFileSize = HEADER_LENGTH + (long) data.length; + + // Resize file + channel.truncate(newFileSize); + channel.position(newFileSize - 1); + channel.write(ByteBuffer.wrap(new byte[] { 0 })); + + // Map and write header + data + mapped = channel.map(FileChannel.MapMode.READ_WRITE, 0, newFileSize); + mapped.position(0); + mapped.put(MAGIC_NUMBER); + mapped.put(FILE_FORMAT_VERSION); + mapped.put(data); + + bitCapacity = newBitCapacity; + } + + /** + * Set/clear a single bit in the mapped bitfield. + * + * Layout is identical to ByteArrayUtil.toByteArray(BitSet): bits are packed 8 per byte, with bit index i at byte (i + * >>> 3), bit (i & 7). + */ + private void setOnDiskBit(int bitIndex, boolean value) { + if (mapped == null || bitIndex < 0) { + return; + } + + int byteIndex = bitIndex >>> 3; + int bitInByte = bitIndex & 7; + + int fileOffset = HEADER_LENGTH + byteIndex; + if (fileOffset >= mapped.capacity()) { + // Should not happen if ensureCapacityForBit() is used correctly + return; + } + + byte b = mapped.get(fileOffset); + int mask = 1 << bitInByte; + + if (value) { + b = (byte) (b | mask); + } else { + b = (byte) (b & ~mask); + } + + mapped.put(fileOffset, b); + } } diff --git a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/btree/ConcurrentNodeCache.java b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/btree/ConcurrentNodeCache.java index bb0f6693a5a..641c9b39526 100644 --- a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/btree/ConcurrentNodeCache.java +++ b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/btree/ConcurrentNodeCache.java @@ -19,8 +19,6 @@ class ConcurrentNodeCache extends ConcurrentCache { - private final static int CONCURRENCY = Runtime.getRuntime().availableProcessors(); - private final Function reader; private static final Consumer writeNode = node -> { @@ -40,7 +38,7 @@ public ConcurrentNodeCache(Function reader) { } public void flush() { - cache.forEachValue(CONCURRENCY, writeNode); + cache.values().forEach(writeNode); } public void put(Node node) { diff --git a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/btree/Node.java b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/btree/Node.java index d6898e8a90b..01e34c1c823 100644 --- a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/btree/Node.java +++ b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/btree/Node.java @@ -12,9 +12,10 @@ import java.io.IOException; import java.nio.ByteBuffer; +import java.util.ArrayList; import java.util.Arrays; -import java.util.Iterator; -import java.util.concurrent.ConcurrentLinkedDeque; +import java.util.Collections; +import java.util.List; import java.util.concurrent.atomic.AtomicInteger; import java.util.function.Function; @@ -52,7 +53,9 @@ class Node { /** * Registered listeners that want to be notified of changes to the node. */ - private final ConcurrentLinkedDeque listeners = new ConcurrentLinkedDeque<>(); + private final Object listenerMutex = new Object(); + + private NodeListenerHandle listenerHead; /** * Creates a new Node object with the specified ID. @@ -104,6 +107,18 @@ public int getUsageCount() { return usageCount.get(); } + int getRegisteredListenerCount() { + synchronized (listenerMutex) { + int count = 0; + for (NodeListenerHandle cursor = listenerHead; cursor != null; cursor = cursor.next) { + if (!cursor.isRemoved()) { + count++; + } + } + return count; + } + } + public boolean dataChanged() { return dataChanged; } @@ -392,14 +407,45 @@ public void rotateRight(int valueIdx, Node leftChildNode, Node rightChildNode) t notifyRotatedRight(valueIdx, leftChildNode, rightChildNode); } - public void register(NodeListener listener) { - // assert !listeners.contains(listener); - listeners.add(listener); + public NodeListenerHandle register(NodeListener listener) { + NodeListenerHandle handle = new NodeListenerHandle(this, listener); + synchronized (listenerMutex) { + handle.next = listenerHead; + if (listenerHead != null) { + listenerHead.prev = handle; + } + listenerHead = handle; + } + return handle; } public void deregister(NodeListener listener) { - // assert listeners.contains(listener); - listeners.removeFirstOccurrence(listener); + NodeListenerHandle handle = null; + synchronized (listenerMutex) { + for (NodeListenerHandle cursor = listenerHead; cursor != null; cursor = cursor.next) { + if (cursor.listener == listener) { + handle = cursor; + break; + } + } + } + if (handle != null) { + handle.remove(); + } + } + + void removeListenerHandle(NodeListenerHandle handle) { + synchronized (listenerMutex) { + if (handle.prev != null) { + handle.prev.next = handle.next; + } else if (listenerHead == handle) { + listenerHead = handle.next; + } + + if (handle.next != null) { + handle.next.prev = handle.prev; + } + } } private void notifyValueAdded(int index) { @@ -436,26 +482,39 @@ private interface NodeListenerNotifier { } private void notifyListeners(NodeListenerNotifier notifier) throws IOException { - Iterator iter = listeners.iterator(); - - while (iter.hasNext()) { - boolean deregister = notifier.apply(iter.next()); - + for (NodeListenerHandle handle : snapshotListeners()) { + if (handle.isRemoved()) { + continue; + } + boolean deregister = notifier.apply(handle.listener); if (deregister) { - iter.remove(); + handle.remove(); } } } private void notifySafeListeners(Function notifier) { - Iterator iter = listeners.iterator(); - - while (iter.hasNext()) { - boolean deregister = notifier.apply(iter.next()); - + for (NodeListenerHandle handle : snapshotListeners()) { + if (handle.isRemoved()) { + continue; + } + boolean deregister = notifier.apply(handle.listener); if (deregister) { - iter.remove(); + handle.remove(); + } + } + } + + private List snapshotListeners() { + synchronized (listenerMutex) { + if (listenerHead == null) { + return Collections.emptyList(); + } + List snapshot = new ArrayList<>(); + for (NodeListenerHandle cursor = listenerHead; cursor != null; cursor = cursor.next) { + snapshot.add(cursor); } + return snapshot; } } diff --git a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/btree/NodeListenerHandle.java b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/btree/NodeListenerHandle.java new file mode 100644 index 00000000000..edfa704bae9 --- /dev/null +++ b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/btree/NodeListenerHandle.java @@ -0,0 +1,37 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.btree; + +import java.util.concurrent.atomic.AtomicBoolean; + +final class NodeListenerHandle { + + final NodeListener listener; + final Node node; + NodeListenerHandle prev; + NodeListenerHandle next; + private final AtomicBoolean removed = new AtomicBoolean(false); + + NodeListenerHandle(Node node, NodeListener listener) { + this.node = node; + this.listener = listener; + } + + boolean isRemoved() { + return removed.get(); + } + + void remove() { + if (removed.compareAndSet(false, true)) { + node.removeListenerHandle(this); + } + } +} diff --git a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/btree/RangeIterator.java b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/btree/RangeIterator.java index e6a6a3847e6..977c9d89c7b 100644 --- a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/btree/RangeIterator.java +++ b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/btree/RangeIterator.java @@ -11,7 +11,8 @@ package org.eclipse.rdf4j.sail.nativerdf.btree; import java.io.IOException; -import java.util.LinkedList; +import java.util.ArrayDeque; +import java.util.Deque; import java.util.concurrent.atomic.AtomicBoolean; import org.eclipse.rdf4j.common.io.ByteArrayUtil; @@ -35,14 +36,11 @@ class RangeIterator implements RecordIterator, NodeListener { private final AtomicBoolean revisitValue = new AtomicBoolean(); /** - * Tracks the parent nodes of {@link #currentNode}. + * Tracks parent nodes, child indices and handles for {@link #currentNode}. */ - private final LinkedList parentNodeStack = new LinkedList<>(); + private final Deque parentStack = new ArrayDeque<>(); - /** - * Tracks the index of child nodes in parent nodes. - */ - private final LinkedList parentIndexStack = new LinkedList<>(); + private NodeListenerHandle currentNodeHandle; private volatile int currentIdx; @@ -97,7 +95,7 @@ private void findMinimum() { return; } - nextCurrentNode.register(this); + currentNodeHandle = nextCurrentNode.register(this); currentIdx = 0; // Search first value >= minValue, or the left-most value in case @@ -173,11 +171,8 @@ public void close() throws IOException { closed = true; tree.btreeLock.readLock().lock(); try { - while (popStacks()) { - } - - assert parentNodeStack.isEmpty(); - assert parentIndexStack.isEmpty(); + clearTraversalState(); + assert parentStack.isEmpty(); } finally { tree.btreeLock.readLock().unlock(); } @@ -187,31 +182,57 @@ public void close() throws IOException { } private void pushStacks(Node newChildNode) { - newChildNode.register(this); - parentNodeStack.add(currentNode); - parentIndexStack.add(currentIdx); + NodeListenerHandle childHandle = newChildNode.register(this); + parentStack.addLast(new StackFrame(currentNode, currentIdx, currentNodeHandle)); currentNode = newChildNode; + currentNodeHandle = childHandle; currentIdx = 0; } private synchronized boolean popStacks() throws IOException { - Node nextCurrentNode = currentNode; - if (nextCurrentNode == null) { - // There's nothing to pop + if (currentNode == null && parentStack.isEmpty()) { return false; } - nextCurrentNode.deregister(this); - nextCurrentNode.release(); - - if (!parentNodeStack.isEmpty()) { - currentNode = parentNodeStack.removeLast(); - currentIdx = parentIndexStack.removeLast(); + releaseCurrentFrame(); + StackFrame previous = parentStack.pollLast(); + if (previous != null) { + currentNode = previous.node; + currentIdx = previous.childIndex; + currentNodeHandle = previous.handle; return true; - } else { - currentNode = null; - currentIdx = 0; - return false; + } + + currentNode = null; + currentIdx = 0; + currentNodeHandle = null; + return false; + } + + private void clearTraversalState() throws IOException { + while (currentNode != null || !parentStack.isEmpty()) { + releaseCurrentFrame(); + StackFrame previous = parentStack.pollLast(); + if (previous == null) { + currentNode = null; + currentIdx = 0; + currentNodeHandle = null; + break; + } + currentNode = previous.node; + currentIdx = previous.childIndex; + currentNodeHandle = previous.handle; + } + } + + private void releaseCurrentFrame() throws IOException { + Node nextCurrentNode = currentNode; + if (nextCurrentNode != null) { + if (currentNodeHandle != null) { + currentNodeHandle.remove(); + currentNodeHandle = null; + } + nextCurrentNode.release(); } } @@ -224,13 +245,11 @@ public boolean valueAdded(Node node, int addedIndex) { currentIdx++; } } else { - for (int i = 0; i < parentNodeStack.size(); i++) { - if (node == parentNodeStack.get(i)) { - int parentIdx = parentIndexStack.get(i); - if (addedIndex < parentIdx) { - parentIndexStack.set(i, parentIdx + 1); + for (StackFrame frame : parentStack) { + if (node == frame.node) { + if (addedIndex < frame.childIndex) { + frame.childIndex++; } - break; } } @@ -248,11 +267,10 @@ public boolean valueRemoved(Node node, int removedIndex) { currentIdx--; } } else { - for (int i = 0; i < parentNodeStack.size(); i++) { - if (node == parentNodeStack.get(i)) { - int parentIdx = parentIndexStack.get(i); - if (removedIndex < parentIdx) { - parentIndexStack.set(i, parentIdx - 1); + for (StackFrame frame : parentStack) { + if (node == frame.node) { + if (removedIndex < frame.childIndex) { + frame.childIndex--; } break; @@ -286,23 +304,24 @@ public boolean rotatedLeft(Node node, int valueIndex, Node leftChildNode, Node r revisitValue.set(true); } } else { - for (int i = 0; i < parentNodeStack.size(); i++) { - Node stackNode = parentNodeStack.get(i); - - if (stackNode == rightChildNode) { - int stackIdx = parentIndexStack.get(i); + for (StackFrame frame : parentStack) { + if (frame.node == rightChildNode) { + int stackIdx = frame.childIndex; if (stackIdx == 0) { - // this node is no longer the parent, replace with left - // sibling - rightChildNode.deregister(this); + // this node is no longer the parent, replace with left sibling + NodeListenerHandle replacedHandle = frame.handle; + if (replacedHandle != null) { + replacedHandle.remove(); + } rightChildNode.release(); leftChildNode.use(); - leftChildNode.register(this); + NodeListenerHandle leftHandle = leftChildNode.register(this); - parentNodeStack.set(i, leftChildNode); - parentIndexStack.set(i, leftChildNode.getValueCount()); + frame.node = leftChildNode; + frame.handle = leftHandle; + frame.childIndex = leftChildNode.getValueCount(); } break; @@ -315,23 +334,24 @@ public boolean rotatedLeft(Node node, int valueIndex, Node leftChildNode, Node r @Override public boolean rotatedRight(Node node, int valueIndex, Node leftChildNode, Node rightChildNode) throws IOException { - for (int i = 0; i < parentNodeStack.size(); i++) { - Node stackNode = parentNodeStack.get(i); - - if (stackNode == leftChildNode) { - int stackIdx = parentIndexStack.get(i); + for (StackFrame frame : parentStack) { + if (frame.node == leftChildNode) { + int stackIdx = frame.childIndex; if (stackIdx == leftChildNode.getValueCount()) { - // this node is no longer the parent, replace with right - // sibling - leftChildNode.deregister(this); + // this node is no longer the parent, replace with right sibling + NodeListenerHandle replacedHandle = frame.handle; + if (replacedHandle != null) { + replacedHandle.remove(); + } leftChildNode.release(); rightChildNode.use(); - rightChildNode.register(this); + NodeListenerHandle rightHandle = rightChildNode.register(this); - parentNodeStack.set(i, rightChildNode); - parentIndexStack.set(i, 0); + frame.node = rightChildNode; + frame.handle = rightHandle; + frame.childIndex = 0; } break; @@ -350,31 +370,40 @@ public boolean nodeSplit(Node node, Node newNode, int medianIdx) throws IOExcept Node nextCurrentNode = currentNode; if (node == nextCurrentNode) { if (currentIdx > medianIdx) { + if (currentNodeHandle != null) { + currentNodeHandle.remove(); + currentNodeHandle = null; + } nextCurrentNode.release(); deregister = true; newNode.use(); - newNode.register(this); + NodeListenerHandle newHandle = newNode.register(this); currentNode = newNode; + currentNodeHandle = newHandle; currentIdx -= medianIdx + 1; } } else { - for (int i = 0; i < parentNodeStack.size(); i++) { - Node parentNode = parentNodeStack.get(i); - - if (node == parentNode) { - int parentIdx = parentIndexStack.get(i); + for (StackFrame frame : parentStack) { + if (node == frame.node) { + int parentIdx = frame.childIndex; if (parentIdx > medianIdx) { + NodeListenerHandle replacedHandle = frame.handle; + if (replacedHandle != null) { + replacedHandle.remove(); + } + Node parentNode = frame.node; parentNode.release(); deregister = true; newNode.use(); - newNode.register(this); + NodeListenerHandle newHandle = newNode.register(this); - parentNodeStack.set(i, newNode); - parentIndexStack.set(i, parentIdx - medianIdx - 1); + frame.node = newNode; + frame.handle = newHandle; + frame.childIndex = parentIdx - medianIdx - 1; } break; @@ -393,27 +422,36 @@ public boolean nodeMergedWith(Node sourceNode, Node targetNode, int mergeIdx) th Node nextCurrentNode = currentNode; if (sourceNode == nextCurrentNode) { + if (currentNodeHandle != null) { + currentNodeHandle.remove(); + currentNodeHandle = null; + } nextCurrentNode.release(); deregister = true; targetNode.use(); - targetNode.register(this); + NodeListenerHandle newHandle = targetNode.register(this); currentNode = targetNode; + currentNodeHandle = newHandle; currentIdx += mergeIdx; } else { - for (int i = 0; i < parentNodeStack.size(); i++) { - Node parentNode = parentNodeStack.get(i); - - if (sourceNode == parentNode) { + for (StackFrame frame : parentStack) { + if (sourceNode == frame.node) { + NodeListenerHandle replacedHandle = frame.handle; + if (replacedHandle != null) { + replacedHandle.remove(); + } + Node parentNode = frame.node; parentNode.release(); deregister = true; targetNode.use(); - targetNode.register(this); + NodeListenerHandle newHandle = targetNode.register(this); - parentNodeStack.set(i, targetNode); - parentIndexStack.set(i, mergeIdx + parentIndexStack.get(i)); + frame.node = targetNode; + frame.handle = newHandle; + frame.childIndex = mergeIdx + frame.childIndex; break; } @@ -429,4 +467,16 @@ public String toString() { "tree=" + tree + '}'; } + + private static final class StackFrame { + Node node; + int childIndex; + NodeListenerHandle handle; + + StackFrame(Node node, int childIndex, NodeListenerHandle handle) { + this.node = node; + this.childIndex = childIndex; + this.handle = handle; + } + } } diff --git a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/config/NativeStoreConfig.java b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/config/NativeStoreConfig.java index bbfd3ce3d58..b757cf0e3f8 100644 --- a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/config/NativeStoreConfig.java +++ b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/config/NativeStoreConfig.java @@ -38,6 +38,26 @@ public class NativeStoreConfig extends BaseSailConfig { private int namespaceCacheSize = -1; private int namespaceIDCacheSize = -1; + // WAL: expose max segment bytes via config (optional) + private long walMaxSegmentBytes = -1L; + + // Additional WAL configuration options + private int walQueueCapacity = -1; + private int walBatchBufferBytes = -1; + private String walSyncPolicy; // expects one of ValueStoreWalConfig.SyncPolicy + private long walSyncIntervalMillis = -1L; + private long walIdlePollIntervalMillis = -1L; + private String walDirectoryName; // relative to dataDir + + // When true, WAL bootstrap runs synchronously during open before accepting new values + private boolean walSyncBootstrapOnOpen = false; + + // When true, reconstruct ValueStore from WAL during open if empty/missing + private boolean walAutoRecoverOnOpen = false; + + // When false, completely disable the ValueStore WAL + private boolean walEnabled = true; + public NativeStoreConfig() { super(NativeStoreFactory.SAIL_TYPE); } @@ -104,6 +124,86 @@ public void setNamespaceIDCacheSize(int namespaceIDCacheSize) { this.namespaceIDCacheSize = namespaceIDCacheSize; } + public long getWalMaxSegmentBytes() { + return walMaxSegmentBytes; + } + + public void setWalMaxSegmentBytes(long walMaxSegmentBytes) { + this.walMaxSegmentBytes = walMaxSegmentBytes; + } + + public int getWalQueueCapacity() { + return walQueueCapacity; + } + + public void setWalQueueCapacity(int walQueueCapacity) { + this.walQueueCapacity = walQueueCapacity; + } + + public int getWalBatchBufferBytes() { + return walBatchBufferBytes; + } + + public void setWalBatchBufferBytes(int walBatchBufferBytes) { + this.walBatchBufferBytes = walBatchBufferBytes; + } + + public String getWalSyncPolicy() { + return walSyncPolicy; + } + + public void setWalSyncPolicy(String walSyncPolicy) { + this.walSyncPolicy = walSyncPolicy; + } + + public long getWalSyncIntervalMillis() { + return walSyncIntervalMillis; + } + + public void setWalSyncIntervalMillis(long walSyncIntervalMillis) { + this.walSyncIntervalMillis = walSyncIntervalMillis; + } + + public long getWalIdlePollIntervalMillis() { + return walIdlePollIntervalMillis; + } + + public void setWalIdlePollIntervalMillis(long walIdlePollIntervalMillis) { + this.walIdlePollIntervalMillis = walIdlePollIntervalMillis; + } + + public String getWalDirectoryName() { + return walDirectoryName; + } + + public void setWalDirectoryName(String walDirectoryName) { + this.walDirectoryName = walDirectoryName; + } + + public boolean getWalSyncBootstrapOnOpen() { + return walSyncBootstrapOnOpen; + } + + public void setWalSyncBootstrapOnOpen(boolean walSyncBootstrapOnOpen) { + this.walSyncBootstrapOnOpen = walSyncBootstrapOnOpen; + } + + public boolean getWalAutoRecoverOnOpen() { + return walAutoRecoverOnOpen; + } + + public void setWalAutoRecoverOnOpen(boolean walAutoRecoverOnOpen) { + this.walAutoRecoverOnOpen = walAutoRecoverOnOpen; + } + + public boolean getWalEnabled() { + return walEnabled; + } + + public void setWalEnabled(boolean walEnabled) { + this.walEnabled = walEnabled; + } + @Override public Resource export(Model m) { if (Configurations.useLegacyConfig()) { @@ -131,6 +231,38 @@ public Resource export(Model m) { if (namespaceIDCacheSize >= 0) { m.add(implNode, CONFIG.Native.namespaceIDCacheSize, literal(namespaceIDCacheSize)); } + // WAL configuration properties + if (walMaxSegmentBytes >= 0) { + m.add(implNode, CONFIG.Native.walMaxSegmentBytes, literal(walMaxSegmentBytes)); + } + if (walQueueCapacity > 0) { + m.add(implNode, CONFIG.Native.walQueueCapacity, literal(walQueueCapacity)); + } + if (walBatchBufferBytes > 0) { + m.add(implNode, CONFIG.Native.walBatchBufferBytes, literal(walBatchBufferBytes)); + } + if (walSyncPolicy != null) { + m.add(implNode, CONFIG.Native.walSyncPolicy, literal(walSyncPolicy)); + } + if (walSyncIntervalMillis >= 0) { + m.add(implNode, CONFIG.Native.walSyncIntervalMillis, literal(walSyncIntervalMillis)); + } + if (walIdlePollIntervalMillis >= 0) { + m.add(implNode, CONFIG.Native.walIdlePollIntervalMillis, literal(walIdlePollIntervalMillis)); + } + if (walDirectoryName != null) { + m.add(implNode, CONFIG.Native.walDirectoryName, literal(walDirectoryName)); + } + // Only export when true to avoid noise + if (walSyncBootstrapOnOpen) { + m.add(implNode, CONFIG.Native.walSyncBootstrapOnOpen, literal(true)); + } + if (walAutoRecoverOnOpen) { + m.add(implNode, CONFIG.Native.walAutoRecoverOnOpen, literal(true)); + } + if (!walEnabled) { + m.add(implNode, CONFIG.Native.walEnabled, literal(false)); + } return implNode; } @@ -157,6 +289,7 @@ private Resource exportLegacy(Model m) { if (namespaceIDCacheSize >= 0) { m.add(implNode, NAMESPACE_ID_CACHE_SIZE, literal(namespaceIDCacheSize)); } + // legacy export does not define a schema term; omit for legacy return implNode; } @@ -224,6 +357,94 @@ public void parse(Model m, Resource implNode) throws SailConfigException { + " property, found " + lit); } }); + + // WAL configuration properties + Configurations.getLiteralValue(m, implNode, CONFIG.Native.walMaxSegmentBytes) + .ifPresent(lit -> { + try { + setWalMaxSegmentBytes(lit.longValue()); + } catch (NumberFormatException e) { + throw new SailConfigException("Long value required for " + + CONFIG.Native.walMaxSegmentBytes + " property, found " + lit); + } + }); + + Configurations.getLiteralValue(m, implNode, CONFIG.Native.walQueueCapacity) + .ifPresent(lit -> { + try { + setWalQueueCapacity(lit.intValue()); + } catch (NumberFormatException e) { + throw new SailConfigException("Integer value required for " + + CONFIG.Native.walQueueCapacity + " property, found " + lit); + } + }); + + Configurations.getLiteralValue(m, implNode, CONFIG.Native.walBatchBufferBytes) + .ifPresent(lit -> { + try { + setWalBatchBufferBytes(lit.intValue()); + } catch (NumberFormatException e) { + throw new SailConfigException("Integer value required for " + + CONFIG.Native.walBatchBufferBytes + " property, found " + lit); + } + }); + + Configurations.getLiteralValue(m, implNode, CONFIG.Native.walSyncPolicy) + .ifPresent(lit -> setWalSyncPolicy(lit.getLabel())); + + Configurations.getLiteralValue(m, implNode, CONFIG.Native.walSyncIntervalMillis) + .ifPresent(lit -> { + try { + setWalSyncIntervalMillis(lit.longValue()); + } catch (NumberFormatException e) { + throw new SailConfigException("Long value required for " + + CONFIG.Native.walSyncIntervalMillis + " property, found " + lit); + } + }); + + Configurations.getLiteralValue(m, implNode, CONFIG.Native.walIdlePollIntervalMillis) + .ifPresent(lit -> { + try { + setWalIdlePollIntervalMillis(lit.longValue()); + } catch (NumberFormatException e) { + throw new SailConfigException("Long value required for " + + CONFIG.Native.walIdlePollIntervalMillis + " property, found " + lit); + } + }); + + Configurations.getLiteralValue(m, implNode, CONFIG.Native.walDirectoryName) + .ifPresent(lit -> setWalDirectoryName(lit.getLabel())); + + Configurations.getLiteralValue(m, implNode, CONFIG.Native.walSyncBootstrapOnOpen) + .ifPresent(lit -> { + try { + setWalSyncBootstrapOnOpen(lit.booleanValue()); + } catch (IllegalArgumentException e) { + throw new SailConfigException("Boolean value required for " + + CONFIG.Native.walSyncBootstrapOnOpen + " property, found " + lit); + } + }); + + Configurations.getLiteralValue(m, implNode, CONFIG.Native.walAutoRecoverOnOpen) + .ifPresent(lit -> { + try { + setWalAutoRecoverOnOpen(lit.booleanValue()); + } catch (IllegalArgumentException e) { + throw new SailConfigException("Boolean value required for " + + CONFIG.Native.walAutoRecoverOnOpen + " property, found " + lit); + } + }); + + Configurations.getLiteralValue(m, implNode, CONFIG.Native.walEnabled) + .ifPresent(lit -> { + try { + setWalEnabled(lit.booleanValue()); + } catch (IllegalArgumentException e) { + throw new SailConfigException( + "Boolean value required for " + CONFIG.Native.walEnabled + " property, found " + + lit); + } + }); } catch (ModelException e) { throw new SailConfigException(e.getMessage(), e); } diff --git a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/config/NativeStoreFactory.java b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/config/NativeStoreFactory.java index 8d1ca19cffc..26c858df305 100644 --- a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/config/NativeStoreFactory.java +++ b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/config/NativeStoreFactory.java @@ -16,6 +16,7 @@ import org.eclipse.rdf4j.sail.config.SailFactory; import org.eclipse.rdf4j.sail.config.SailImplConfig; import org.eclipse.rdf4j.sail.nativerdf.NativeStore; +import org.eclipse.rdf4j.sail.nativerdf.wal.ValueStoreWalConfig; /** * A {@link SailFactory} that creates {@link NativeStore}s based on RDF configuration data. @@ -75,6 +76,39 @@ public Sail getSail(SailImplConfig config) throws SailConfigException { nativeStore.setIterationCacheSyncThreshold(nativeConfig.getIterationCacheSyncThreshold()); } + if (nativeConfig.getWalMaxSegmentBytes() > 0) { + nativeStore.setWalMaxSegmentBytes(nativeConfig.getWalMaxSegmentBytes()); + } + + if (nativeConfig.getWalQueueCapacity() > 0) { + nativeStore.setWalQueueCapacity(nativeConfig.getWalQueueCapacity()); + } + if (nativeConfig.getWalBatchBufferBytes() > 0) { + nativeStore.setWalBatchBufferBytes(nativeConfig.getWalBatchBufferBytes()); + } + if (nativeConfig.getWalSyncPolicy() != null) { + try { + nativeStore.setWalSyncPolicy(ValueStoreWalConfig.SyncPolicy + .valueOf(nativeConfig.getWalSyncPolicy().toUpperCase())); + } catch (IllegalArgumentException e) { + throw new SailConfigException("Invalid walSyncPolicy: " + nativeConfig.getWalSyncPolicy()); + } + } + if (nativeConfig.getWalSyncIntervalMillis() >= 0) { + nativeStore.setWalSyncIntervalMillis(nativeConfig.getWalSyncIntervalMillis()); + } + if (nativeConfig.getWalIdlePollIntervalMillis() >= 0) { + nativeStore.setWalIdlePollIntervalMillis(nativeConfig.getWalIdlePollIntervalMillis()); + } + if (nativeConfig.getWalDirectoryName() != null) { + nativeStore.setWalDirectoryName(nativeConfig.getWalDirectoryName()); + } + // New: allow configuring synchronous WAL bootstrap during open + nativeStore.setWalSyncBootstrapOnOpen(nativeConfig.getWalSyncBootstrapOnOpen()); + // New: allow configuring auto-recovery of ValueStore from WAL during open + nativeStore.setWalAutoRecoverOnOpen(nativeConfig.getWalAutoRecoverOnOpen()); + nativeStore.setWalEnabled(nativeConfig.getWalEnabled()); + EvaluationStrategyFactory evalStratFactory = nativeConfig.getEvaluationStrategyFactory(); if (evalStratFactory != null) { nativeStore.setEvaluationStrategyFactory(evalStratFactory); diff --git a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/datastore/DataStore.java b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/datastore/DataStore.java index 90c46561c40..a648c97f2d5 100644 --- a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/datastore/DataStore.java +++ b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/datastore/DataStore.java @@ -145,9 +145,9 @@ private byte[] attemptToRecoverCorruptData(int id, long offset, byte[] data) thr try { if (valueStore != null && Thread.currentThread().getStackTrace().length < 512) { NativeValue nativeValue = valueStore.data2value(prev, prevData); - logger.warn("Data in previous ID ({}) is: {}", prev, nativeValue); + logger.debug("Data in previous ID ({}) is: {}", prev, nativeValue); } else { - logger.warn("Data in previous ID ({}) is: {}", prev, + logger.debug("Data in previous ID ({}) is: {}", prev, new String(prevData, StandardCharsets.UTF_8)); } } catch (Exception ignored) { @@ -184,9 +184,9 @@ private byte[] attemptToRecoverCorruptData(int id, long offset, byte[] data) thr try { if (valueStore != null && Thread.currentThread().getStackTrace().length < 512) { NativeValue nativeValue = valueStore.data2value(next, nextData); - logger.warn("Data in next ID ({}) is: {}", next, nativeValue); + logger.debug("Data in next ID ({}) is: {}", next, nativeValue); } else { - logger.warn("Data in next ID ({}) is: {}", next, + logger.debug("Data in next ID ({}) is: {}", next, new String(nextData, StandardCharsets.UTF_8)); } } catch (Exception ignored) { diff --git a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/datastore/HashFile.java b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/datastore/HashFile.java index cdb17e02a3b..417de6392e1 100644 --- a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/datastore/HashFile.java +++ b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/datastore/HashFile.java @@ -264,7 +264,9 @@ private void storeID(long bucketOffset, int hash, int id) throws IOException { public void clear() throws IOException { structureLock.writeLock().lock(); - poorMansBloomFilter.clear(); + if (poorMansBloomFilter != null) { + poorMansBloomFilter.clear(); + } try { // Truncate the file to remove any overflow buffers nioFile.truncate(HEADER_LENGTH + (long) bucketCount * recordSize); diff --git a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/model/CorruptValue.java b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/model/CorruptValue.java index db4c1834bdb..f127c255b96 100644 --- a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/model/CorruptValue.java +++ b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/model/CorruptValue.java @@ -32,6 +32,7 @@ public class CorruptValue implements NativeValue { private final byte[] data; private volatile ValueStoreRevision revision; private volatile int internalID; + private transient NativeValue recovered; // optional recovered value constructed from WAL public CorruptValue(ValueStoreRevision revision, int internalID, byte[] data) { setInternalID(internalID, revision); @@ -68,6 +69,21 @@ public byte[] getData() { return data; } + /** + * Set a recovered value corresponding to this corrupt entry. The recovered value should be a NativeValue with its + * internal ID set to the same ID as this corrupt value. + */ + public void setRecovered(NativeValue recovered) { + this.recovered = recovered; + } + + /** + * Returns a recovered value if one was attached; may be null if recovery failed. + */ + public NativeValue getRecovered() { + return recovered; + } + @Override public boolean equals(Object o) { if (this == o) { diff --git a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWAL.java b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWAL.java new file mode 100644 index 00000000000..7f92aa67c06 --- /dev/null +++ b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWAL.java @@ -0,0 +1,943 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import java.io.BufferedInputStream; +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.io.InputStream; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.channels.Channels; +import java.nio.channels.ClosedChannelException; +import java.nio.channels.FileChannel; +import java.nio.channels.FileLock; +import java.nio.file.Files; +import java.nio.file.OpenOption; +import java.nio.file.Path; +import java.nio.file.StandardOpenOption; +import java.time.Instant; +import java.util.List; +import java.util.Objects; +import java.util.concurrent.ArrayBlockingQueue; +import java.util.concurrent.BlockingQueue; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicLong; +import java.util.regex.Matcher; +import java.util.regex.Pattern; +import java.util.stream.Collectors; +import java.util.zip.CRC32; +import java.util.zip.CRC32C; +import java.util.zip.GZIPInputStream; +import java.util.zip.GZIPOutputStream; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.fasterxml.jackson.core.JsonFactory; +import com.fasterxml.jackson.core.JsonGenerator; +import com.fasterxml.jackson.core.JsonParser; +import com.fasterxml.jackson.core.JsonToken; + +/** + * Write-ahead log (WAL) for the ValueStore. The WAL records minted values in append-only segments so they can be + * recovered or searched independently from the on-disk ValueStore files. This class is thread-safe for concurrent + * producers and uses a background writer thread to serialize and fsync according to the configured + * {@link ValueStoreWalConfig.SyncPolicy}. + */ +public final class ValueStoreWAL implements AutoCloseable { + + private static final Logger logger = LoggerFactory.getLogger(ValueStoreWAL.class); + + public BlockingQueue getQueue() { + var queue = this.queue; + if (queue != null) { + return queue; + } + + synchronized (this) { + queue = this.queue; + if (queue == null) { + queue = new ArrayBlockingQueue<>(config.queueCapacity()); + this.queue = queue; + } + return queue; + } + } + + @FunctionalInterface + interface FileChannelOpener { + FileChannel open(Path path, OpenOption... options) throws IOException; + } + + private static final FileChannelOpener DEFAULT_CHANNEL_OPENER = FileChannel::open; + private static volatile FileChannelOpener channelOpener = DEFAULT_CHANNEL_OPENER; + + public static final long NO_LSN = -1L; + + static final Pattern SEGMENT_PATTERN = Pattern.compile("wal-(\\d+)\\.v1(?:\\.gz)?"); + public static final int MAX_FRAME_BYTES = 512 * 1024 * 1024; // 512 MiB safety cap + + private final ValueStoreWalConfig config; + private volatile BlockingQueue queue; + private final AtomicLong nextLsn = new AtomicLong(); + private final AtomicLong lastAppendedLsn = new AtomicLong(NO_LSN); + private final AtomicLong lastForcedLsn = new AtomicLong(NO_LSN); + private final AtomicLong requestedForceLsn = new AtomicLong(NO_LSN); + + private final Object ackMonitor = new Object(); + + private final LogWriter logWriter; + private final Thread writerThread; + + private volatile boolean closed; + private volatile Throwable writerFailure; + + // Reset/purge coordination + private volatile boolean purgeRequested; + private final Object purgeMonitor = new Object(); + private volatile boolean purgeInProgress; + + private final FileChannel lockChannel; + private final FileLock directoryLock; + + private final boolean initialSegmentsPresent; + private final int initialMaxSegmentSeq; + + static void setChannelOpenerForTesting(FileChannelOpener opener) { + channelOpener = opener != null ? opener : DEFAULT_CHANNEL_OPENER; + } + + static void resetChannelOpenerForTesting() { + channelOpener = DEFAULT_CHANNEL_OPENER; + } + + private static FileChannel openWalChannel(Path path, OpenOption... options) throws IOException { + return channelOpener.open(path, options); + } + + private ValueStoreWAL(ValueStoreWalConfig config) throws IOException { + this.config = Objects.requireNonNull(config, "config"); + if (!Files.isDirectory(config.walDirectory())) { + Files.createDirectories(config.walDirectory()); + } + + Path lockFile = config.walDirectory().resolve("lock"); + lockChannel = FileChannel.open(lockFile, StandardOpenOption.CREATE, StandardOpenOption.WRITE); + try { + directoryLock = lockChannel.tryLock(); + } catch (IOException e) { + lockChannel.close(); + throw e; + } + if (directoryLock == null) { + throw new IOException("WAL directory is already locked: " + config.walDirectory()); + } + + DirectoryState state = analyzeDirectory(config.walDirectory()); + this.initialSegmentsPresent = state.hasSegments; + this.initialMaxSegmentSeq = state.maxSequence; + // Seed next LSN from existing WAL, if any, to ensure monotonic LSNs across restarts + if (initialSegmentsPresent) { + try (ValueStoreWalReader reader = ValueStoreWalReader.open(config)) { + var it = reader.iterator(); + while (it.hasNext()) { + it.next(); + } + long last = reader.lastValidLsn(); + if (last > NO_LSN) { + nextLsn.set(last); + } + } + } + this.logWriter = new LogWriter(initialMaxSegmentSeq); + this.writerThread = new Thread(logWriter, "ValueStoreWalWriter-" + config.storeUuid()); + this.writerThread.setDaemon(true); + this.writerThread.start(); + } + + /** + * Open a ValueStore WAL for the provided configuration. The WAL directory is created if it does not already exist. + * If existing segments are detected, the next LSN is seeded from the last valid record to ensure monotonicity + * across restarts. + */ + public static ValueStoreWAL open(ValueStoreWalConfig config) throws IOException { + return new ValueStoreWAL(config); + } + + public ValueStoreWalConfig config() { + return config; + } + + /** + * Append a minted value record to the WAL. + * + * @param id the ValueStore internal id + * @param kind the kind of value (IRI, BNODE, LITERAL, NAMESPACE) + * @param lexical the lexical form (may be empty but never null) + * @param datatype the datatype IRI string for literals, otherwise empty + * @param language the language tag for literals, otherwise empty + * @param hash a hash of the underlying serialized value + * @return the log sequence number (LSN) assigned to the record + */ + public long logMint(int id, ValueStoreWalValueKind kind, String lexical, String datatype, String language, int hash) + throws IOException { + ensureOpen(); + long lsn = nextLsn.incrementAndGet(); + ValueStoreWalRecord record = new ValueStoreWalRecord(lsn, id, kind, lexical, datatype, language, hash); + enqueue(record); + return lsn; + } + + /** + * Block until the given LSN is durably forced to disk according to the configured sync policy. This is a no-op when + * {@code lsn <= NO_LSN} or after the WAL is closed. + */ + public void awaitDurable(long lsn) throws InterruptedException, IOException { + if (lsn <= NO_LSN || closed) { + return; + } + ensureOpen(); + if (lastForcedLsn.get() >= lsn) { + return; + } + requestForce(lsn); + + // fsync is slow, so when using the INTERVAL sync policy we won't wait for fsync to finish + if (config.syncPolicy() == ValueStoreWalConfig.SyncPolicy.INTERVAL) { + return; + } + synchronized (ackMonitor) { + while (lastForcedLsn.get() < lsn && writerFailure == null && !closed) { + ackMonitor.wait(TimeUnit.MILLISECONDS.toMillis(10)); + } + } + if (writerFailure != null) { + throw propagate(writerFailure); + } + } + + /** + * Returns {@code true} if WAL segments were already present in the directory when this WAL was opened. + */ + public boolean hasInitialSegments() { + return initialSegmentsPresent; + } + + /** + * Returns {@code true} once {@link #close()} has been invoked and the writer thread has terminated. + */ + public boolean isClosed() { + return closed; + } + + /** + * Purges all existing WAL segments from the WAL directory. This is used when the associated ValueStore is cleared, + * to ensure that a subsequent WAL recovery cannot resurrect deleted values. + *

    + * The purge is coordinated with the writer thread: the current segment (if any) is closed before files are deleted, + * and the writer is reset to create a fresh segment on the next append. + */ + /** + * Purge all WAL segments from the WAL directory. Coordinated with the writer thread to close the current segment + * before deletion and reset to a fresh segment after purge completes. + */ + public void purgeAllSegments() throws IOException { + ensureOpen(); + // Signal the writer to perform a coordinated purge and wait for completion + synchronized (purgeMonitor) { + purgeRequested = true; + purgeInProgress = true; + purgeMonitor.notifyAll(); + long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(10); + while (purgeInProgress && writerFailure == null && !closed) { + long remaining = deadline - System.nanoTime(); + if (remaining <= 0) { + throw new IOException("Timed out waiting for WAL purge to complete"); + } + try { + purgeMonitor.wait(Math.min(TimeUnit.NANOSECONDS.toMillis(remaining), 50)); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new IOException("Interrupted while waiting for WAL purge", e); + } + } + if (writerFailure != null) { + throw propagate(writerFailure); + } + if (closed) { + throw new IOException("WAL is closed"); + } + } + } + + @Override + public void close() throws IOException { + if (closed) { + return; + } + closed = true; + logWriter.shutdown(); + try { + writerThread.join(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + try { + logWriter.close(); + } finally { + try { + if (directoryLock != null && directoryLock.isValid()) { + directoryLock.release(); + } + } finally { + if (lockChannel != null && lockChannel.isOpen()) { + lockChannel.close(); + } + } + } + if (writerFailure != null) { + throw propagate(writerFailure); + } + } + + private void requestForce(long lsn) { + requestedForceLsn.updateAndGet(prev -> Math.max(prev, lsn)); + } + + private void enqueue(ValueStoreWalRecord record) throws IOException { + boolean offered = false; + int spins = 0; + while (!offered) { + offered = getQueue().offer(record); + if (!offered) { + if (spins < 100) { + Thread.onSpinWait(); + spins++; + } else { + try { + getQueue().put(record); + offered = true; + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new IOException("Interrupted while enqueueing WAL record", e); + } + } + } + } + } + + private void ensureOpen() throws IOException { + if (closed) { + throw new IOException("WAL is closed"); + } + if (writerFailure != null) { + throw propagate(writerFailure); + } + } + + private IOException propagate(Throwable throwable) { + if (throwable instanceof IOException) { + return (IOException) throwable; + } + return new IOException("WAL writer failure", throwable); + } + + private DirectoryState analyzeDirectory(Path walDirectory) throws IOException { + if (!Files.isDirectory(walDirectory)) { + return new DirectoryState(false, 0); + } + int maxSequence = 0; + boolean hasSegments = false; + List paths; + try (var stream = Files.list(walDirectory)) { + paths = stream.collect(Collectors.toList()); + } + for (Path path : paths) { + Matcher matcher = SEGMENT_PATTERN.matcher(path.getFileName().toString()); + if (matcher.matches()) { + hasSegments = true; + try { + int segment = readSegmentSequence(path); + if (segment > maxSequence) { + maxSequence = segment; + } + } catch (IOException e) { + logger.warn("Failed to read WAL segment header for {}", path.getFileName(), e); + } + } + } + return new DirectoryState(hasSegments, maxSequence); + } + + static int readSegmentSequence(Path path) throws IOException { + boolean compressed = path.getFileName().toString().endsWith(".gz"); + try (var rawIn = new BufferedInputStream(Files.newInputStream(path)); + InputStream in = compressed ? new GZIPInputStream(rawIn) : rawIn) { + byte[] lenBytes = in.readNBytes(4); + if (lenBytes.length < 4) { + return 0; + } + ByteBuffer lenBuf = ByteBuffer.wrap(lenBytes).order(ByteOrder.LITTLE_ENDIAN); + int frameLen = lenBuf.getInt(); + if (frameLen <= 0) { + return 0; + } + byte[] jsonBytes = in.readNBytes(frameLen); + if (jsonBytes.length < frameLen) { + return 0; + } + // skip CRC + in.readNBytes(4); + JsonFactory factory = new JsonFactory(); + try (JsonParser parser = factory.createParser(jsonBytes)) { + while (parser.nextToken() != JsonToken.END_OBJECT) { + if (parser.currentToken() == JsonToken.FIELD_NAME) { + String field = parser.getCurrentName(); + parser.nextToken(); + if ("segment".equals(field)) { + return parser.getIntValue(); + } + } + } + } + } + return 0; + } + + private static final class DirectoryState { + final boolean hasSegments; + final int maxSequence; + + DirectoryState(boolean hasSegments, int maxSequence) { + this.hasSegments = hasSegments; + this.maxSequence = maxSequence; + } + } + + private final class LogWriter implements Runnable { + + private final CRC32C crc32c = new CRC32C(); + private final int batchSize; + private FileChannel segmentChannel; + private Path segmentPath; + private int segmentSequence; + private long segmentBytes; + private int segmentLastMintedId; + private int segmentFirstMintedId; + private volatile ByteBuffer ioBuffer; + // Reuse JSON infrastructure to reduce allocations per record + private final JsonFactory jsonFactory = new JsonFactory(); + private final ReusableByteArrayOutputStream jsonBuffer = new ReusableByteArrayOutputStream(256); + private volatile boolean running = true; + + LogWriter(int existingSegments) { + this.segmentSequence = existingSegments; + this.batchSize = config.batchBufferBytes(); + this.segmentChannel = null; + this.segmentPath = null; + this.segmentBytes = 0L; + this.segmentLastMintedId = 0; + this.segmentFirstMintedId = 0; + } + + private ByteBuffer getIoBuffer() { + if (ioBuffer == null) { + synchronized (this) { + if (ioBuffer == null) { + ioBuffer = ByteBuffer.allocateDirect(batchSize).order(ByteOrder.LITTLE_ENDIAN); + } + } + } + return ioBuffer; + } + + @Override + public void run() { + try { + long lastSyncCheck = System.nanoTime(); + while (running || !getQueue().isEmpty()) { + // Handle purge requests promptly + if (purgeRequested) { + performPurgeInternal(); + } + ValueStoreWalRecord record; + try { + record = getQueue().poll(config.idlePollInterval().toNanos(), TimeUnit.NANOSECONDS); + } catch (InterruptedException e) { + if (!running) { + break; + } + continue; + } + if (record != null) { + append(record); + } + + boolean pendingForce = requestedForceLsn.get() > NO_LSN + && requestedForceLsn.get() > lastForcedLsn.get(); + boolean syncIntervalElapsed = config.syncPolicy() == ValueStoreWalConfig.SyncPolicy.INTERVAL + && System.nanoTime() - lastSyncCheck >= config.syncInterval().toNanos(); + if (record == null) { + if (pendingForce || config.syncPolicy() == ValueStoreWalConfig.SyncPolicy.ALWAYS + || syncIntervalElapsed) { + flushAndForce(); + lastSyncCheck = System.nanoTime(); + } + } else if (config.syncPolicy() == ValueStoreWalConfig.SyncPolicy.ALWAYS) { + flushAndForce(); + lastSyncCheck = System.nanoTime(); + } else if (pendingForce && requestedForceLsn.get() <= lastAppendedLsn.get()) { + flushAndForce(); + lastSyncCheck = System.nanoTime(); + } + } + flushAndForce(); + } catch (Throwable t) { + writerFailure = t; + } finally { + try { + flushAndForce(); + } catch (Throwable t) { + writerFailure = t; + } + closeQuietly(segmentChannel); + synchronized (ackMonitor) { + ackMonitor.notifyAll(); + } + } + } + + void shutdown() { + running = false; + } + + void close() throws IOException { + closeQuietly(segmentChannel); + } + + private void ensureSegmentWritable() throws IOException { + if (segmentPath == null || segmentChannel == null) { + return; + } + if (Files.exists(segmentPath)) { + return; + } + if (config.syncPolicy() == ValueStoreWalConfig.SyncPolicy.ALWAYS) { + throw new IOException("Current WAL segment has been removed: " + segmentPath); + } + logger.error("Detected deletion of active WAL segment {}; continuing with a new segment", + segmentPath.getFileName()); + ByteBuffer pending = null; + if (getIoBuffer().position() > 0) { + ByteBuffer duplicate = getIoBuffer().duplicate(); + duplicate.flip(); + if (duplicate.hasRemaining()) { + pending = ByteBuffer.allocate(duplicate.remaining()); + pending.put(duplicate); + pending.flip(); + } + } + getIoBuffer().clear(); + closeQuietly(segmentChannel); + int previousFirstId = segmentFirstMintedId; + int previousLastId = segmentLastMintedId; + segmentChannel = null; + segmentPath = null; + segmentBytes = 0L; + segmentFirstMintedId = 0; + if (previousFirstId > 0) { + startSegment(previousFirstId, false); + segmentLastMintedId = previousLastId; + if (pending != null) { + while (pending.hasRemaining()) { + segmentChannel.write(pending); + } + segmentBytes += pending.limit(); + } + } else { + segmentLastMintedId = previousLastId; + } + } + + private void append(ValueStoreWalRecord record) throws IOException { + ensureSegmentWritable(); + if (segmentChannel == null) { + startSegment(record.id()); + } + // Encode JSON for the record into reusable buffer without copying + int jsonLength = encodeIntoReusableBuffer(record); + int framedLength = 4 + jsonLength + 4; + if (segmentBytes + framedLength > config.maxSegmentBytes()) { + flushBuffer(); + finishCurrentSegment(); + startSegment(record.id()); + } + // Write header length (4 bytes) + if (getIoBuffer().remaining() < 4) { + flushBuffer(); + } + getIoBuffer().putInt(jsonLength); + + // Write JSON payload in chunks to avoid BufferOverflowException + int offset = 0; + byte[] jsonBytes = jsonBuffer.buffer(); + while (offset < jsonLength) { + if (getIoBuffer().remaining() == 0) { + flushBuffer(); + } + int toWrite = Math.min(getIoBuffer().remaining(), jsonLength - offset); + getIoBuffer().put(jsonBytes, offset, toWrite); + offset += toWrite; + } + + // Write CRC (4 bytes) + int crc = checksum(jsonBytes, jsonLength); + if (getIoBuffer().remaining() < 4) { + flushBuffer(); + } + getIoBuffer().putInt(crc); + + segmentBytes += framedLength; + if (record.id() > segmentLastMintedId) { + segmentLastMintedId = record.id(); + } + lastAppendedLsn.set(record.lsn()); + } + + private void performPurgeInternal() { + try { + // Ensure any buffered data is not left around; close current segment + closeQuietly(segmentChannel); + // Drop any frames that were queued prior to purge using dequeue semantics to ensure + // any producers blocked in queue.put() are signalled via notFull. + while (getQueue().poll() != null) { + // intentionally empty: draining via poll() triggers the normal signalling path + } + getIoBuffer().clear(); + // Delete all existing segments from disk + deleteAllSegments(); + // Reset writer state so the next append starts a fresh segment + segmentPath = null; + segmentChannel = null; + segmentBytes = 0L; + segmentFirstMintedId = 0; + segmentLastMintedId = 0; + } catch (IOException e) { + writerFailure = e; + } finally { + purgeRequested = false; + synchronized (purgeMonitor) { + purgeInProgress = false; + purgeMonitor.notifyAll(); + } + } + } + + private void flushAndForce() throws IOException { + flushAndForce(false); + } + + private void flushAndForce(boolean forceEvenForInterval) throws IOException { + if (lastAppendedLsn.get() <= lastForcedLsn.get()) { + return; + } + flushBuffer(); + if (segmentChannel != null && segmentChannel.isOpen()) { + try { + boolean shouldForce = forceEvenForInterval + || config.syncPolicy() != ValueStoreWalConfig.SyncPolicy.INTERVAL; + if (shouldForce) { + segmentChannel.force(false); + if (segmentPath != null) { + ValueStoreWalDebug.fireForceEvent(segmentPath); + } + } + } catch (ClosedChannelException e) { + // ignore; channel already closed during shutdown + } + } + long forced = lastAppendedLsn.get(); + lastForcedLsn.set(forced); + // Clear pending force request without dropping newer requests that may arrive concurrently. + // Use CAS to only clear if the observed value is still <= forced; if another thread published + // a higher LSN in the meantime, we must not overwrite it with NO_LSN. + long cur = requestedForceLsn.get(); + while (cur != NO_LSN && cur <= forced) { + if (requestedForceLsn.compareAndSet(cur, NO_LSN)) { + break; + } + cur = requestedForceLsn.get(); + } + synchronized (ackMonitor) { + ackMonitor.notifyAll(); + } + } + + private void flushBuffer() throws IOException { + ensureSegmentWritable(); + if (segmentChannel == null) { + getIoBuffer().clear(); + return; + } + getIoBuffer().flip(); + while (getIoBuffer().hasRemaining()) { + segmentChannel.write(getIoBuffer()); + } + getIoBuffer().clear(); + } + + private void finishCurrentSegment() throws IOException { + if (segmentChannel == null) { + return; + } + boolean forceInterval = config.syncPolicy() == ValueStoreWalConfig.SyncPolicy.INTERVAL; + flushAndForce(forceInterval); + int summaryLastId = segmentLastMintedId; + Path toCompress = segmentPath; + closeQuietly(segmentChannel); + segmentChannel = null; + segmentPath = null; + segmentBytes = 0L; + segmentFirstMintedId = 0; + segmentLastMintedId = 0; + if (toCompress != null) { + gzipAndDelete(toCompress, summaryLastId); + } + } + + /** + * Rotate the current WAL segment. This is a small wrapper used by tests to ensure that rotation forces the + * previous segment to disk before closing it. New segments will be started lazily on the next append. + */ + @SuppressWarnings("unused") + private void rotateSegment() throws IOException { + finishCurrentSegment(); + } + + private void startSegment(int firstId) throws IOException { + startSegment(firstId, true); + } + + private void startSegment(int firstId, boolean incrementSequence) throws IOException { + if (incrementSequence) { + segmentSequence++; + } + segmentPath = config.walDirectory().resolve(buildSegmentFileName(firstId)); + if (Files.exists(segmentPath)) { + logger.warn("Overwriting existing WAL segment {}", segmentPath.getFileName()); + } + segmentChannel = openWalChannel(segmentPath, StandardOpenOption.CREATE, StandardOpenOption.WRITE, + StandardOpenOption.TRUNCATE_EXISTING); + segmentBytes = 0L; + segmentFirstMintedId = firstId; + segmentLastMintedId = 0; + writeHeader(firstId); + } + + private String buildSegmentFileName(int firstId) { + return "wal-" + firstId + ".v1"; + } + + private void gzipAndDelete(Path src, int lastMintedId) { + Path gz = src.resolveSibling(src.getFileName().toString() + ".gz"); + long srcSize; + try { + srcSize = Files.size(src); + } catch (IOException e) { + // If we can't stat the file, don't attempt compression + logger.warn("Skipping compression of WAL segment {} because it is no longer accessible", + src.getFileName()); + return; + } + int summaryFrameLength; + CRC32 crc32 = new CRC32(); + try (var in = Files.newInputStream(src); + FileChannel gzChannel = openWalChannel(gz, StandardOpenOption.CREATE, StandardOpenOption.WRITE, + StandardOpenOption.TRUNCATE_EXISTING); + GZIPOutputStream gzOut = new GZIPOutputStream(Channels.newOutputStream(gzChannel))) { + byte[] buf = new byte[1 << 16]; + int r; + while ((r = in.read(buf)) >= 0) { + gzOut.write(buf, 0, r); + crc32.update(buf, 0, r); + } + byte[] summaryFrame = buildSummaryFrame(lastMintedId, crc32.getValue()); + summaryFrameLength = summaryFrame.length; + gzOut.write(summaryFrame); + gzOut.finish(); + gzOut.flush(); + gzChannel.force(false); + ValueStoreWalDebug.fireForceEvent(gz); + } catch (IOException e) { + // Compression failed: do not delete original; clean up partial gzip if present + logger.warn("Failed to compress WAL segment {}: {}", src.getFileName(), e.getMessage()); + try { + Files.deleteIfExists(gz); + } catch (IOException ignore) { + } + return; + } + // Verify gzip contains full original data plus summary by reading back and counting bytes + long decompressedBytes = 0L; + byte[] verifyBuf = new byte[1 << 16]; + try (var gin = new GZIPInputStream(Files.newInputStream(gz))) { + int r; + while ((r = gin.read(verifyBuf)) >= 0) { + decompressedBytes += r; + } + } catch (IOException e) { + logger.warn("Failed to verify compressed WAL segment {}: {}", gz.getFileName(), e.getMessage()); + try { + Files.deleteIfExists(gz); + } catch (IOException ignore) { + } + return; + } + if (decompressedBytes != srcSize + summaryFrameLength) { + // Verification failed: keep original, remove corrupt gzip + try { + Files.deleteIfExists(gz); + } catch (IOException ignore) { + } + return; + } + try { + Files.deleteIfExists(src); + } catch (IOException e) { + logger.warn("Failed to delete WAL segment {} after compression: {}", src.getFileName(), e.getMessage()); + } + } + + private byte[] buildSummaryFrame(int lastMintedId, long crc32Value) throws IOException { + JsonFactory factory = new JsonFactory(); + ByteArrayOutputStream baos = new ByteArrayOutputStream(128); + try (JsonGenerator gen = factory.createGenerator(baos)) { + gen.writeStartObject(); + gen.writeStringField("t", "S"); + gen.writeNumberField("lastId", lastMintedId); + gen.writeNumberField("crc32", crc32Value & 0xFFFFFFFFL); + gen.writeEndObject(); + } + baos.write('\n'); + byte[] jsonBytes = baos.toByteArray(); + ByteBuffer buffer = ByteBuffer.allocate(4 + jsonBytes.length + 4).order(ByteOrder.LITTLE_ENDIAN); + buffer.putInt(jsonBytes.length); + buffer.put(jsonBytes); + int crc = checksum(jsonBytes); + buffer.putInt(crc); + buffer.flip(); + byte[] framed = new byte[buffer.remaining()]; + buffer.get(framed); + return framed; + } + + private void writeHeader(int firstId) throws IOException { + JsonFactory factory = new JsonFactory(); + ByteArrayOutputStream baos = new ByteArrayOutputStream(256); + try (JsonGenerator gen = factory.createGenerator(baos)) { + gen.writeStartObject(); + gen.writeStringField("t", "V"); + gen.writeNumberField("ver", 1); + gen.writeStringField("store", config.storeUuid()); + gen.writeStringField("engine", "valuestore"); + gen.writeNumberField("created", Instant.now().getEpochSecond()); + gen.writeNumberField("segment", segmentSequence); + gen.writeNumberField("firstId", firstId); + gen.writeEndObject(); + } + // NDJSON: newline-delimited JSON + baos.write('\n'); + byte[] jsonBytes = baos.toByteArray(); + ByteBuffer buffer = ByteBuffer.allocate(4 + jsonBytes.length + 4).order(ByteOrder.LITTLE_ENDIAN); + buffer.putInt(jsonBytes.length); + buffer.put(jsonBytes); + int crc = checksum(jsonBytes); + buffer.putInt(crc); + buffer.flip(); + while (buffer.hasRemaining()) { + segmentChannel.write(buffer); + } + segmentBytes += buffer.limit(); + } + + private int checksum(byte[] data) { + return checksum(data, data.length); + } + + private int checksum(byte[] data, int len) { + crc32c.reset(); + crc32c.update(data, 0, len); + return (int) crc32c.getValue(); + } + + private int encodeIntoReusableBuffer(ValueStoreWalRecord record) throws IOException { + jsonBuffer.reset(); + try (JsonGenerator gen = jsonFactory.createGenerator(jsonBuffer)) { + gen.writeStartObject(); + gen.writeStringField("t", "M"); + gen.writeNumberField("lsn", record.lsn()); + gen.writeNumberField("id", record.id()); + gen.writeStringField("vk", String.valueOf(record.valueKind().code())); + gen.writeStringField("lex", record.lexical() == null ? "" : record.lexical()); + gen.writeStringField("dt", record.datatype() == null ? "" : record.datatype()); + gen.writeStringField("lang", record.language() == null ? "" : record.language()); + gen.writeNumberField("hash", record.hash()); + gen.writeEndObject(); + } + jsonBuffer.write('\n'); // NDJSON newline + return jsonBuffer.size(); + } + + private void closeQuietly(FileChannel channel) { + if (channel != null) { + try { + channel.close(); + } catch (IOException ignore) { + // ignore + } + } + } + + // Minimal extension to access internal buffer without copying + private final class ReusableByteArrayOutputStream extends ByteArrayOutputStream { + ReusableByteArrayOutputStream(int size) { + super(size); + } + + byte[] buffer() { + return this.buf; + } + } + } + + private void deleteAllSegments() throws IOException { + List toDelete; + try (var stream = Files.list(config.walDirectory())) { + toDelete = stream + .filter(Files::isRegularFile) + .filter(path -> { + String name = path.getFileName().toString(); + return name.matches("wal-[0-9]+\\.v1") || name.matches("wal-[0-9]+\\.v1\\.gz"); + }) + .collect(Collectors.toList()); + } + for (Path p : toDelete) { + try { + Files.deleteIfExists(p); + } catch (IOException e) { + logger.warn("Failed to delete WAL segment {}", p.getFileName(), e); + throw e; + } + } + } + +} diff --git a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalConfig.java b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalConfig.java new file mode 100644 index 00000000000..08d734814d7 --- /dev/null +++ b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalConfig.java @@ -0,0 +1,235 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import java.nio.file.Path; +import java.time.Duration; +import java.util.Objects; + +/** + * Configuration for the ValueStore WAL implementation. + */ +public final class ValueStoreWalConfig { + + public static final String DEFAULT_DIRECTORY_NAME = "value-store-wal"; + + /** + * Controls when the WAL writer flushes buffered frames to disk and when it invokes + * {@link java.nio.channels.FileChannel#force(boolean)} to guarantee durability. Choose the policy that matches the + * desired durability vs. throughput trade-off. + */ + public enum SyncPolicy { + /** + * Forces the WAL after every append (and whenever the writer wakes up without new work). This delivers the + * strongest durability and detects deleted segments immediately, at the cost of running + * {@code FileChannel.force} for every minted value. + */ + ALWAYS, + /** + * Flushes buffered frames to the WAL file whenever the configured {@link ValueStoreWalConfig#syncInterval()} + * elapses without new work but never calls {@code FileChannel.force}. Even + * {@link org.eclipse.rdf4j.sail.nativerdf.ValueStore#awaitWalDurable(long)} only waits for frames to leave the + * in-memory queue, so crashes can drop recently minted values. Choose this for maximum throughput with + * best-effort persistence. + */ + INTERVAL, + /** + * Leaves frames queued until {@link org.eclipse.rdf4j.sail.nativerdf.ValueStore#awaitWalDurable(long)} (invoked + * during NativeStore commit) requests a force. This keeps ingestion fast during a transaction but issues + * {@code FileChannel.force} when the transaction commits, providing durability at commit boundaries only. + */ + COMMIT + } + + private final Path walDirectory; + private final Path snapshotsDirectory; + private final String storeUuid; + private final long maxSegmentBytes; + private final int queueCapacity; + private final int batchBufferBytes; + private final SyncPolicy syncPolicy; + private final Duration syncInterval; + private final Duration idlePollInterval; + private final boolean syncBootstrapOnOpen; + private final boolean recoverValueStoreOnOpen; + + private ValueStoreWalConfig(Builder builder) { + this.walDirectory = builder.walDirectory; + this.snapshotsDirectory = builder.snapshotsDirectory; + this.storeUuid = builder.storeUuid; + this.maxSegmentBytes = builder.maxSegmentBytes; + this.queueCapacity = builder.queueCapacity; + this.batchBufferBytes = builder.batchBufferBytes; + this.syncPolicy = builder.syncPolicy; + this.syncInterval = builder.syncInterval; + this.idlePollInterval = builder.idlePollInterval; + this.syncBootstrapOnOpen = builder.syncBootstrapOnOpen; + this.recoverValueStoreOnOpen = builder.recoverValueStoreOnOpen; + } + + public Path walDirectory() { + return walDirectory; + } + + public Path snapshotsDirectory() { + return snapshotsDirectory; + } + + public String storeUuid() { + return storeUuid; + } + + public long maxSegmentBytes() { + return maxSegmentBytes; + } + + public int queueCapacity() { + return queueCapacity; + } + + public int batchBufferBytes() { + return batchBufferBytes; + } + + public SyncPolicy syncPolicy() { + return syncPolicy; + } + + public Duration syncInterval() { + return syncInterval; + } + + public Duration idlePollInterval() { + return idlePollInterval; + } + + /** + * When true, the ValueStore will synchronously rebuild the WAL from existing values during open before allowing any + * new values to be added. When false (default), bootstrap runs asynchronously in the background. + */ + public boolean syncBootstrapOnOpen() { + return syncBootstrapOnOpen; + } + + /** + * When true, the ValueStore will attempt to reconstruct missing or empty ValueStore files from the WAL during open + * before allowing any operations. + */ + public boolean recoverValueStoreOnOpen() { + return recoverValueStoreOnOpen; + } + + public static Builder builder() { + return new Builder(); + } + + public static final class Builder { + + private Path walDirectory; + private Path snapshotsDirectory; + private String storeUuid; + private long maxSegmentBytes = 128 * 1024 * 1024; // 128 MB + private int queueCapacity = 16 * 1024; + private int batchBufferBytes = 128 * 1024; // 128 KB + private SyncPolicy syncPolicy = SyncPolicy.INTERVAL; + private Duration syncInterval = Duration.ofSeconds(1); + private Duration idlePollInterval = Duration.ofMillis(100); + private boolean syncBootstrapOnOpen = false; + private boolean recoverValueStoreOnOpen = false; + + private Builder() { + } + + public Builder walDirectory(Path walDirectory) { + this.walDirectory = Objects.requireNonNull(walDirectory, "walDirectory"); + if (this.snapshotsDirectory == null) { + this.snapshotsDirectory = walDirectory.resolve("snapshots"); + } + return this; + } + + public Builder snapshotsDirectory(Path snapshotsDirectory) { + this.snapshotsDirectory = Objects.requireNonNull(snapshotsDirectory, "snapshotsDirectory"); + return this; + } + + public Builder storeUuid(String storeUuid) { + this.storeUuid = Objects.requireNonNull(storeUuid, "storeUuid"); + return this; + } + + public Builder maxSegmentBytes(long maxSegmentBytes) { + this.maxSegmentBytes = maxSegmentBytes; + return this; + } + + public Builder queueCapacity(int queueCapacity) { + this.queueCapacity = queueCapacity; + return this; + } + + public Builder batchBufferBytes(int batchBufferBytes) { + this.batchBufferBytes = batchBufferBytes; + return this; + } + + public Builder syncPolicy(SyncPolicy syncPolicy) { + this.syncPolicy = Objects.requireNonNull(syncPolicy, "syncPolicy"); + return this; + } + + public Builder syncInterval(Duration syncInterval) { + this.syncInterval = Objects.requireNonNull(syncInterval, "syncInterval"); + return this; + } + + public Builder idlePollInterval(Duration idlePollInterval) { + this.idlePollInterval = Objects.requireNonNull(idlePollInterval, "idlePollInterval"); + return this; + } + + /** + * Control whether WAL bootstrap happens synchronously during open. Default is false. + */ + public Builder syncBootstrapOnOpen(boolean syncBootstrapOnOpen) { + this.syncBootstrapOnOpen = syncBootstrapOnOpen; + return this; + } + + /** Enable automatic ValueStore recovery from WAL during open. */ + public Builder recoverValueStoreOnOpen(boolean recoverValueStoreOnOpen) { + this.recoverValueStoreOnOpen = recoverValueStoreOnOpen; + return this; + } + + public ValueStoreWalConfig build() { + if (walDirectory == null) { + throw new IllegalStateException("walDirectory must be set"); + } + if (snapshotsDirectory == null) { + snapshotsDirectory = walDirectory.resolve("snapshots"); + } + if (storeUuid == null || storeUuid.isEmpty()) { + throw new IllegalStateException("storeUuid must be set"); + } + if (maxSegmentBytes <= 0) { + throw new IllegalStateException("maxSegmentBytes must be positive"); + } + if (queueCapacity <= 0) { + throw new IllegalStateException("queueCapacity must be positive"); + } + if (batchBufferBytes <= 4096) { + throw new IllegalStateException("batchBufferBytes must be > 4KB"); + } + return new ValueStoreWalConfig(this); + } + } +} diff --git a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalDebug.java b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalDebug.java new file mode 100644 index 00000000000..ba4aadb09ae --- /dev/null +++ b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalDebug.java @@ -0,0 +1,41 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import java.nio.file.Path; +import java.util.Objects; +import java.util.function.Consumer; + +/** + * Package-private debug hook that allows tests to observe when WAL files are forced to disk. + */ +final class ValueStoreWalDebug { + + private static volatile Consumer forceListener; + + private ValueStoreWalDebug() { + } + + static void setForceListener(Consumer listener) { + forceListener = listener; + } + + static void clearForceListener() { + forceListener = null; + } + + static void fireForceEvent(Path path) { + Consumer listener = forceListener; + if (listener != null) { + listener.accept(Objects.requireNonNull(path, "path")); + } + } +} diff --git a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReader.java b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReader.java new file mode 100644 index 00000000000..ed199fc55e7 --- /dev/null +++ b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReader.java @@ -0,0 +1,522 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import java.io.IOException; +import java.io.InputStream; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.channels.FileChannel; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.StandardOpenOption; +import java.util.ArrayList; +import java.util.Comparator; +import java.util.Iterator; +import java.util.List; +import java.util.NoSuchElementException; +import java.util.Objects; +import java.util.regex.Pattern; +import java.util.zip.CRC32; +import java.util.zip.CRC32C; +import java.util.zip.GZIPInputStream; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.fasterxml.jackson.core.JsonFactory; +import com.fasterxml.jackson.core.JsonParser; +import com.fasterxml.jackson.core.JsonToken; + +/** + * Reader for ValueStore WAL segments that yields minted records in LSN order across segments. It tolerates truncated or + * missing tail data by stopping at the last valid record observed. Completeness can be queried via + * {@link #isComplete()} and by inspecting {@link ScanResult#complete()}. + */ +public final class ValueStoreWalReader implements AutoCloseable { + + private static final Pattern SEGMENT_PATTERN = Pattern.compile("wal-(\\d+)\\.v1(?:\\.gz)?"); + private static final Logger logger = LoggerFactory.getLogger(ValueStoreWalReader.class); + + private final ValueStoreWalConfig config; + private final JsonFactory jsonFactory = new JsonFactory(); + // Streaming iteration state + private final List segments; + private int segIndex = -1; + private FileChannel channel; + private GZIPInputStream gzIn; + private boolean stop; + private boolean eos; // end-of-segment indicator for current stream + private long lastValidLsn = ValueStoreWAL.NO_LSN; + private final boolean missingSegments; + private boolean summaryMissing; + private boolean currentSegmentCompressed; + private boolean currentSegmentSummarySeen; + // CRC32 of the original (uncompressed) segment contents, accumulated while reading a compressed segment. + private CRC32 currentSegmentCrc32; + + private ValueStoreWalReader(ValueStoreWalConfig config) { + this.config = Objects.requireNonNull(config, "config"); + List segs; + try { + segs = listSegments(); + } catch (IOException e) { + segs = List.of(); + } + this.segments = segs; + this.missingSegments = hasSequenceGaps(segs); + this.summaryMissing = false; + this.currentSegmentCompressed = false; + this.currentSegmentSummarySeen = false; + } + + /** + * Create a reader for the given configuration. No I/O is performed until iteration begins. + */ + public static ValueStoreWalReader open(ValueStoreWalConfig config) { + return new ValueStoreWalReader(config); + } + + /** + * Scan the WAL and return all minted records together with bookkeeping about last valid LSN and completeness. + */ + public ScanResult scan() throws IOException { + List records = new ArrayList<>(); + Iterator it = this.iterator(); + while (it.hasNext()) { + records.add(it.next()); + } + return new ScanResult(records, this.lastValidLsn(), this.isComplete()); + } + + /** On-demand iterator over minted WAL records. */ + public Iterator iterator() { + return new RecordIterator(); + } + + /** Highest valid LSN observed during reading (iterator/scan). */ + public long lastValidLsn() { + return lastValidLsn; + } + + // Iterator utils: open/close segments and read single records + private boolean openNextSegment() throws IOException { + closeCurrentSegment(); + segIndex++; + if (segIndex >= segments.size()) { + return false; + } + SegmentEntry entry = segments.get(segIndex); + Path p = entry.path; + currentSegmentCompressed = entry.compressed; + currentSegmentSummarySeen = false; + if (currentSegmentCompressed) { + gzIn = new GZIPInputStream(Files.newInputStream(p)); + channel = null; + currentSegmentCrc32 = new CRC32(); + } else { + channel = FileChannel.open(p, StandardOpenOption.READ); + gzIn = null; + currentSegmentCrc32 = null; + } + return true; + } + + private void closeCurrentSegment() throws IOException { + if (currentSegmentCompressed && !currentSegmentSummarySeen) { + summaryMissing = true; + } + currentSegmentCrc32 = null; + if (channel != null && channel.isOpen()) { + channel.close(); + } + channel = null; + if (gzIn != null) { + gzIn.close(); + } + gzIn = null; + eos = false; + currentSegmentCompressed = false; + currentSegmentSummarySeen = false; + } + + private static int readIntLE(InputStream in) throws IOException { + byte[] b = in.readNBytes(4); + if (b.length < 4) { + return -1; + } + return ((b[0] & 0xFF)) | ((b[1] & 0xFF) << 8) | ((b[2] & 0xFF) << 16) | ((b[3] & 0xFF) << 24); + } + + private static class Item { + final Path path; + final long firstId; + final int sequence; + final boolean compressed; + + Item(Path path, long firstId, int sequence, boolean compressed) { + this.path = path; + this.firstId = firstId; + this.sequence = sequence; + this.compressed = compressed; + } + } + + private List listSegments() throws IOException { + + List items = new ArrayList<>(); + if (!Files.isDirectory(config.walDirectory())) { + return List.of(); + } + try (var stream = Files.list(config.walDirectory())) { + stream.forEach(p -> { + var m = SEGMENT_PATTERN.matcher(p.getFileName().toString()); + if (m.matches()) { + long firstId = Long.parseLong(m.group(1)); + boolean compressed = p.getFileName().toString().endsWith(".gz"); + int sequence = 0; + try { + sequence = ValueStoreWAL.readSegmentSequence(p); + } catch (IOException e) { + logger.warn("Failed to read WAL segment header for {}", p.getFileName(), e); + } + items.add(new Item(p, firstId, sequence, compressed)); + } + }); + } + items.sort(Comparator.comparingInt(it -> it.sequence)); + List segments = new ArrayList<>(items.size()); + for (Item it : items) { + segments.add(new SegmentEntry(it.path, it.firstId, it.sequence, it.compressed)); + } + return segments; + } + + private boolean hasSequenceGaps(List entries) { + if (entries.isEmpty()) { + return false; + } + int expected = entries.get(0).sequence; + if (expected > 1) { + return true; + } + for (SegmentEntry entry : entries) { + if (entry.sequence != expected) { + return true; + } + expected++; + } + return false; + } + + private ValueStoreWalRecord readOneFromChannel() throws IOException { + ByteBuffer header = ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN); + header.clear(); + int read = channel.read(header); + if (read == -1) { + eos = true; + return null; // clean end of segment + } + if (read < 4) { + stop = true; // truncated header + return null; + } + header.flip(); + int length = header.getInt(); + if (length <= 0 || (long) length > ValueStoreWAL.MAX_FRAME_BYTES) { + stop = true; + return null; + } + byte[] data = new byte[length]; + ByteBuffer dataBuf = ByteBuffer.wrap(data); + int total = 0; + while (total < length) { + int n = channel.read(dataBuf); + if (n < 0) { + stop = true; // truncated record + return null; + } + total += n; + } + ByteBuffer crcBuf = ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN); + int crcRead = channel.read(crcBuf); + if (crcRead < 4) { + stop = true; + return null; + } + crcBuf.flip(); + int expectedCrc = crcBuf.getInt(); + CRC32C crc32c = new CRC32C(); + crc32c.update(data, 0, data.length); + if ((int) crc32c.getValue() != expectedCrc) { + stop = true; + return null; + } + Parsed parsed = parseJson(data); + if (parsed.type == 'M') { + ValueStoreWalRecord r = new ValueStoreWalRecord(parsed.lsn, parsed.id, parsed.kind, parsed.lex, parsed.dt, + parsed.lang, + parsed.hash); + lastValidLsn = r.lsn(); + return r; + } + if (parsed.lsn > lastValidLsn) { + lastValidLsn = parsed.lsn; + } + // non-minted record within segment; continue reading same segment + eos = false; + return null; + } + + private ValueStoreWalRecord readOneFromGzip() throws IOException { + int length = readIntLE(gzIn); + if (length == -1) { + eos = true; + return null; // end of stream cleanly + } + if (length <= 0 || (long) length > ValueStoreWAL.MAX_FRAME_BYTES) { + stop = true; + return null; + } + byte[] data = gzIn.readNBytes(length); + if (data.length < length) { + stop = true; // truncated + return null; + } + int expectedCrc = readIntLE(gzIn); + CRC32C crc32c = new CRC32C(); + crc32c.update(data, 0, data.length); + if ((int) crc32c.getValue() != expectedCrc) { + stop = true; + return null; + } + Parsed parsed = parseJson(data); + // For compressed segments, accumulate CRC32 over the original segment bytes (lenLE + data + crcLE) + if (currentSegmentCrc32 != null && parsed.type != 'S') { + // length in little-endian + ByteBuffer lenBuf = ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putInt(length); + lenBuf.flip(); + currentSegmentCrc32.update(lenBuf.array(), 0, 4); + currentSegmentCrc32.update(data, 0, data.length); + ByteBuffer crcBuf = ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putInt(expectedCrc); + crcBuf.flip(); + currentSegmentCrc32.update(crcBuf.array(), 0, 4); + } + if (parsed.type == 'M') { + ValueStoreWalRecord r = new ValueStoreWalRecord(parsed.lsn, parsed.id, parsed.kind, parsed.lex, parsed.dt, + parsed.lang, + parsed.hash); + lastValidLsn = r.lsn(); + return r; + } + if (parsed.type == 'S') { + currentSegmentSummarySeen = true; + // Validate CRC32 of segment contents against summary + if (currentSegmentCrc32 != null) { + long computed = currentSegmentCrc32.getValue() & 0xFFFFFFFFL; + if (parsed.summaryCrc32 != computed) { + // mark stream as invalid/incomplete + stop = true; + } + } + } + if (parsed.lsn > lastValidLsn) { + lastValidLsn = parsed.lsn; + } + // non-minted record within segment; keep reading + eos = false; + return null; + } + + private final class RecordIterator implements Iterator { + private ValueStoreWalRecord next; + private boolean prepared; + + @Override + public boolean hasNext() { + if (prepared) { + return next != null; + } + try { + prepareNext(); + } catch (IOException e) { + stop = true; + next = null; + } + prepared = true; + return next != null; + } + + @Override + public ValueStoreWalRecord next() { + if (!hasNext()) { + throw new NoSuchElementException(); + } + prepared = false; + ValueStoreWalRecord r = next; + next = null; + return r; + } + + private void prepareNext() throws IOException { + next = null; + if (stop) { + return; + } + while (true) { + if (channel == null && gzIn == null) { + if (!openNextSegment()) { + return; // no more segments + } + } + if (gzIn != null) { + ValueStoreWalRecord r = readOneFromGzip(); + if (r != null) { + next = r; + return; + } + if (stop) { + return; + } + if (eos) { + closeCurrentSegment(); + } + continue; + } + if (channel != null) { + ValueStoreWalRecord r = readOneFromChannel(); + if (r != null) { + next = r; + return; + } + if (stop) { + return; + } + if (eos) { + closeCurrentSegment(); + } + } + } + } + } + + private Parsed parseJson(byte[] jsonBytes) throws IOException { + Parsed parsed = new Parsed(); + try (JsonParser jp = jsonFactory.createParser(jsonBytes)) { + if (jp.nextToken() != JsonToken.START_OBJECT) { + return parsed; + } + while (jp.nextToken() != JsonToken.END_OBJECT) { + String field = jp.getCurrentName(); + jp.nextToken(); + if ("t".equals(field)) { + String t = jp.getValueAsString(""); + parsed.type = t.isEmpty() ? '?' : t.charAt(0); + } else if ("lsn".equals(field)) { + parsed.lsn = jp.getValueAsLong(ValueStoreWAL.NO_LSN); + } else if ("id".equals(field)) { + parsed.id = jp.getValueAsInt(0); + } else if ("lastId".equals(field)) { + parsed.id = jp.getValueAsInt(0); + } else if ("vk".equals(field)) { + String code = jp.getValueAsString(""); + parsed.kind = ValueStoreWalValueKind.fromCode(code); + } else if ("lex".equals(field)) { + parsed.lex = jp.getValueAsString(""); + } else if ("dt".equals(field)) { + parsed.dt = jp.getValueAsString(""); + } else if ("lang".equals(field)) { + parsed.lang = jp.getValueAsString(""); + } else if ("hash".equals(field)) { + parsed.hash = jp.getValueAsInt(0); + } else if ("crc32".equals(field)) { + parsed.summaryCrc32 = jp.getValueAsLong(0L); + } else { + jp.skipChildren(); + } + } + } + return parsed; + } + + private static final class SegmentEntry { + final Path path; + final long firstId; + final int sequence; + final boolean compressed; + + SegmentEntry(Path path, long firstId, int sequence, boolean compressed) { + this.path = path; + this.firstId = firstId; + this.sequence = sequence; + this.compressed = compressed; + } + } + + private static final class Parsed { + char type = '?'; + long lsn = ValueStoreWAL.NO_LSN; + int id = 0; + ValueStoreWalValueKind kind = ValueStoreWalValueKind.NAMESPACE; + String lex = ""; + String dt = ""; + String lang = ""; + int hash = 0; + long summaryCrc32 = 0L; + } + + @Override + public void close() { + try { + closeCurrentSegment(); + } catch (IOException e) { + // ignore on close + } + } + + /** + * Whether the reader observed a complete, contiguous sequence of segments and a valid summary for compressed + * segments, and did not encounter validation errors. + */ + boolean isComplete() { + return !missingSegments && !summaryMissing && !stop; + } + + /** Result of a full WAL scan. */ + public static final class ScanResult { + private final List records; + private final long lastValidLsn; + private final boolean complete; + + public ScanResult(List records, long lastValidLsn, boolean complete) { + this.records = List.copyOf(records); + this.lastValidLsn = lastValidLsn; + this.complete = complete; + } + + /** + * All minted records encountered, in LSN order. + */ + public List records() { + return records; + } + + /** Highest valid LSN observed during the scan. */ + public long lastValidLsn() { + return lastValidLsn; + } + + /** Whether the scan covered a complete and validated set of segments. */ + public boolean complete() { + return complete; + } + } +} diff --git a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalRecord.java b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalRecord.java new file mode 100644 index 00000000000..6765941dd4f --- /dev/null +++ b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalRecord.java @@ -0,0 +1,66 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import java.util.Objects; + +/** + * Representation of a single ValueStore WAL record describing a minted value. + */ +public final class ValueStoreWalRecord { + + private final long lsn; + private final int id; + private final ValueStoreWalValueKind valueKind; + private final String lexical; + private final String datatype; + private final String language; + private final int hash; + + public ValueStoreWalRecord(long lsn, int id, ValueStoreWalValueKind valueKind, String lexical, String datatype, + String language, int hash) { + this.lsn = lsn; + this.id = id; + this.valueKind = Objects.requireNonNull(valueKind, "valueKind"); + this.lexical = lexical == null ? "" : lexical; + this.datatype = datatype == null ? "" : datatype; + this.language = language == null ? "" : language; + this.hash = hash; + } + + public long lsn() { + return lsn; + } + + public int id() { + return id; + } + + public ValueStoreWalValueKind valueKind() { + return valueKind; + } + + public String lexical() { + return lexical; + } + + public String datatype() { + return datatype; + } + + public String language() { + return language; + } + + public int hash() { + return hash; + } +} diff --git a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalRecovery.java b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalRecovery.java new file mode 100644 index 00000000000..f912c7400e5 --- /dev/null +++ b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalRecovery.java @@ -0,0 +1,51 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import java.io.IOException; +import java.util.Collections; +import java.util.LinkedHashMap; +import java.util.Map; + +public final class ValueStoreWalRecovery { + + public Map replay(ValueStoreWalReader reader) throws IOException { + return replayWithReport(reader).dictionary(); + } + + public ReplayReport replayWithReport(ValueStoreWalReader reader) throws IOException { + ValueStoreWalReader.ScanResult scan = reader.scan(); + Map dictionary = new LinkedHashMap<>(); + for (ValueStoreWalRecord record : scan.records()) { + dictionary.putIfAbsent(record.id(), record); + } + return new ReplayReport(dictionary, scan.complete()); + } + + public static final class ReplayReport { + private final Map dictionary; + private final boolean complete; + + public ReplayReport(Map dictionary, boolean complete) { + this.dictionary = Collections + .unmodifiableMap(new LinkedHashMap<>(dictionary)); + this.complete = complete; + } + + public Map dictionary() { + return dictionary; + } + + public boolean complete() { + return complete; + } + } +} diff --git a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalSearch.java b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalSearch.java new file mode 100644 index 00000000000..b426bc20fb8 --- /dev/null +++ b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalSearch.java @@ -0,0 +1,327 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ + +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import java.io.IOException; +import java.io.InputStream; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.channels.FileChannel; +import java.nio.file.Files; +import java.nio.file.NoSuchFileException; +import java.nio.file.Path; +import java.nio.file.StandardOpenOption; +import java.util.ArrayList; +import java.util.List; +import java.util.Objects; +import java.util.Optional; +import java.util.regex.Pattern; +import java.util.zip.CRC32C; +import java.util.zip.GZIPInputStream; + +import org.eclipse.rdf4j.model.Value; +import org.eclipse.rdf4j.model.impl.SimpleValueFactory; + +import com.fasterxml.jackson.core.JsonFactory; +import com.fasterxml.jackson.core.JsonParser; +import com.fasterxml.jackson.core.JsonToken; + +/** + * Utility to search a ValueStore WAL for a specific minted value ID efficiently. + * + * Strategy: scan the first minted record in each segment to determine the best candidate segment (binary search on the + * first IDs), then scan only that segment to locate the requested ID. + */ +public final class ValueStoreWalSearch { + + private static final Pattern SEGMENT_PATTERN = Pattern.compile("wal-(\\d+)\\.v1(?:\\.gz)?"); + + private final ValueStoreWalConfig config; + private final JsonFactory jsonFactory = new JsonFactory(); + private volatile List cachedSegments; + + private ValueStoreWalSearch(ValueStoreWalConfig config) { + this.config = Objects.requireNonNull(config, "config"); + } + + public static ValueStoreWalSearch open(ValueStoreWalConfig config) { + return new ValueStoreWalSearch(config); + } + + /** + * Find and reconstruct a {@link org.eclipse.rdf4j.model.Value} by its ValueStore id using WAL contents only. + * + * @return the reconstructed value if present; {@code null} otherwise + */ + public Value findValueById(int id) throws IOException { + if (!Files.isDirectory(config.walDirectory())) { + invalidateSegmentCache(); + return null; + } + + LookupOutcome firstAttempt = locateCandidate(id, false); + if (firstAttempt.value != null || !firstAttempt.retry) { + return firstAttempt.value; + } + + LookupOutcome secondAttempt = locateCandidate(id, true); + return secondAttempt.value; + } + + private static final class SegFirst { + final Path path; + final int firstId; + + SegFirst(Path p, int id) { + this.path = p; + this.firstId = id; + } + } + + private LookupOutcome locateCandidate(int targetId, boolean forceRefresh) throws IOException { + List segments = loadSegments(forceRefresh); + if (segments.isEmpty()) { + return LookupOutcome.miss(!forceRefresh); + } + + SegFirst candidate = selectSegment(segments, targetId); + if (candidate == null) { + return LookupOutcome.miss(!forceRefresh); + } + + Optional value; + try { + value = scanSegmentForId(candidate.path, targetId); + } catch (NoSuchFileException missingSegment) { + invalidateSegmentCache(); + return LookupOutcome.miss(!forceRefresh); + } + if (value.isPresent()) { + return LookupOutcome.hit(value.get()); + } + return LookupOutcome.miss(!forceRefresh); + } + + private List loadSegments(boolean forceRefresh) throws IOException { + if (forceRefresh) { + invalidateSegmentCache(); + } + + List snapshot = cachedSegments; + if (snapshot != null) { + return snapshot; + } + synchronized (this) { + snapshot = cachedSegments; + if (snapshot == null) { + snapshot = readSegmentsFromDisk(); + cachedSegments = snapshot; + } + return snapshot; + } + } + + private List readSegmentsFromDisk() throws IOException { + if (!Files.isDirectory(config.walDirectory())) { + return List.of(); + } + List segments = new ArrayList<>(); + try (var stream = Files.list(config.walDirectory())) { + stream.forEach(p -> { + var m = SEGMENT_PATTERN.matcher(p.getFileName().toString()); + if (m.matches()) { + long firstId1 = Long.parseLong(m.group(1)); + if (firstId1 >= Integer.MIN_VALUE && firstId1 <= Integer.MAX_VALUE) { + segments.add(new SegFirst(p, (int) firstId1)); + } + } + }); + } + return List.copyOf(segments); + } + + private SegFirst selectSegment(List segments, int targetId) { + SegFirst best = null; + for (SegFirst segment : segments) { + if (segment.firstId > targetId) { + continue; + } + if (best == null || segment.firstId > best.firstId) { + best = segment; + } + } + return best; + } + + private void invalidateSegmentCache() { + cachedSegments = null; + } + + private static final class LookupOutcome { + final Value value; + final boolean retry; + + private LookupOutcome(Value value, boolean retry) { + this.value = value; + this.retry = retry; + } + + static LookupOutcome hit(Value value) { + return new LookupOutcome(value, false); + } + + static LookupOutcome miss(boolean retry) { + return new LookupOutcome(null, retry); + } + } + + private Optional scanSegmentForId(Path segment, int targetId) throws IOException { + if (segment.getFileName().toString().endsWith(".gz")) { + try (GZIPInputStream in = new GZIPInputStream(Files.newInputStream(segment))) { + while (true) { + int length = readIntLE(in); + if (length == -1) + return Optional.empty(); + if (length <= 0 || (long) length > ValueStoreWAL.MAX_FRAME_BYTES) + return Optional.empty(); + byte[] data = in.readNBytes(length); + if (data.length < length) + return Optional.empty(); + int expectedCrc = readIntLE(in); + CRC32C crc32c = new CRC32C(); + crc32c.update(data, 0, data.length); + if ((int) crc32c.getValue() != expectedCrc) + return Optional.empty(); + Parsed p = parseJson(data); + if (p.type == 'M' && p.id == targetId) { + Value value = toValue(p); + if (value != null) { + return Optional.of(value); + } + } + } + } + } + try (FileChannel ch = FileChannel.open(segment, StandardOpenOption.READ)) { + ByteBuffer header = ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN); + while (true) { + header.clear(); + int r = ch.read(header); + if (r == -1) + return Optional.empty(); + if (r < 4) + return Optional.empty(); + header.flip(); + int length = header.getInt(); + if (length <= 0 || (long) length > ValueStoreWAL.MAX_FRAME_BYTES) + return Optional.empty(); + byte[] data = new byte[length]; + ByteBuffer dataBuf = ByteBuffer.wrap(data); + int total = 0; + while (total < length) { + int n = ch.read(dataBuf); + if (n < 0) + return Optional.empty(); + total += n; + } + ByteBuffer crcBuf = ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN); + int crcRead = ch.read(crcBuf); + if (crcRead < 4) + return Optional.empty(); + crcBuf.flip(); + int expectedCrc = crcBuf.getInt(); + CRC32C crc32c = new CRC32C(); + crc32c.update(data, 0, data.length); + if ((int) crc32c.getValue() != expectedCrc) + return Optional.empty(); + Parsed p = parseJson(data); + if (p.type == 'M' && p.id == targetId) { + Value value = toValue(p); + if (value != null) { + return Optional.of(value); + } + } + } + } + } + + private int readIntLE(InputStream in) throws IOException { + byte[] b = in.readNBytes(4); + if (b.length < 4) + return -1; + return (b[0] & 0xFF) | ((b[1] & 0xFF) << 8) | ((b[2] & 0xFF) << 16) | ((b[3] & 0xFF) << 24); + } + + private Parsed parseJson(byte[] jsonBytes) throws IOException { + Parsed parsed = new Parsed(); + try (JsonParser jp = jsonFactory.createParser(jsonBytes)) { + if (jp.nextToken() != JsonToken.START_OBJECT) { + return parsed; + } + while (jp.nextToken() != JsonToken.END_OBJECT) { + String field = jp.getCurrentName(); + jp.nextToken(); + if ("t".equals(field)) { + String t = jp.getValueAsString(""); + parsed.type = t.isEmpty() ? '?' : t.charAt(0); + } else if ("lsn".equals(field)) { + parsed.lsn = jp.getValueAsLong(ValueStoreWAL.NO_LSN); + } else if ("id".equals(field)) { + parsed.id = jp.getValueAsInt(0); + } else if ("vk".equals(field)) { + String code = jp.getValueAsString(""); + parsed.kind = ValueStoreWalValueKind.fromCode(code); + } else if ("lex".equals(field)) { + parsed.lex = jp.getValueAsString(""); + } else if ("dt".equals(field)) { + parsed.dt = jp.getValueAsString(""); + } else if ("lang".equals(field)) { + parsed.lang = jp.getValueAsString(""); + } else if ("hash".equals(field)) { + parsed.hash = jp.getValueAsInt(0); + } else { + jp.skipChildren(); + } + } + } + return parsed; + } + + private Value toValue(Parsed p) { + var vf = SimpleValueFactory.getInstance(); + switch (p.kind) { + case IRI: + return vf.createIRI(p.lex); + case BNODE: + return vf.createBNode(p.lex); + case LITERAL: + if (p.lang != null && !p.lang.isEmpty()) + return vf.createLiteral(p.lex, p.lang); + if (p.dt != null && !p.dt.isEmpty()) + return vf.createLiteral(p.lex, vf.createIRI(p.dt)); + return vf.createLiteral(p.lex); + default: + return null; + } + } + + private static final class Parsed { + char type = '?'; + long lsn = ValueStoreWAL.NO_LSN; + int id = 0; + ValueStoreWalValueKind kind = ValueStoreWalValueKind.NAMESPACE; + String lex = ""; + String dt = ""; + String lang = ""; + int hash = 0; + } +} diff --git a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalValueKind.java b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalValueKind.java new file mode 100644 index 00000000000..3ad0ae0977b --- /dev/null +++ b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalValueKind.java @@ -0,0 +1,46 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ + +package org.eclipse.rdf4j.sail.nativerdf.wal; + +/** + * Enumeration of value kinds that may be persisted in the value store WAL. + */ +public enum ValueStoreWalValueKind { + + IRI('I'), + BNODE('B'), + LITERAL('L'), + NAMESPACE('N'); + + private final char code; + + ValueStoreWalValueKind(char code) { + this.code = code; + } + + public char code() { + return code; + } + + public static ValueStoreWalValueKind fromCode(String code) { + if (code == null || code.isEmpty()) { + throw new IllegalArgumentException("Missing value kind code"); + } + char c = code.charAt(0); + for (ValueStoreWalValueKind kind : values()) { + if (kind.code == c) { + return kind; + } + } + throw new IllegalArgumentException("Unknown value kind code: " + code); + } +} diff --git a/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/wal/package-info.java b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/wal/package-info.java new file mode 100644 index 00000000000..cb57eba8b09 --- /dev/null +++ b/core/sail/nativerdf/src/main/java/org/eclipse/rdf4j/sail/nativerdf/wal/package-info.java @@ -0,0 +1,24 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ + +/** + * ValueStore-specific write-ahead log infrastructure for the NativeStore. These utilities are not intended for general + * NativeStore WAL support. + * + * @apiNote This package is experimental: its existence, signature or behavior may change without warning from one + * release to the next. + */ + +@Experimental + +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import org.eclipse.rdf4j.common.annotation.Experimental; diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/ContextStoreTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/ContextStoreTest.java index 3decf6afa2a..c6befd8a9a5 100644 --- a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/ContextStoreTest.java +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/ContextStoreTest.java @@ -42,7 +42,7 @@ public class ContextStoreTest { File dataDir; /** - * @throws java.lang.Exception + * @throws Exception */ @BeforeEach public void setUp() throws Exception { diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/MemoryMappedTxnStatusFileConfigTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/MemoryMappedTxnStatusFileConfigTest.java new file mode 100644 index 00000000000..edc9441c130 --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/MemoryMappedTxnStatusFileConfigTest.java @@ -0,0 +1,72 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.io.File; + +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +/** + * Verifies that the implementation used for the transaction status file can be controlled via a system property. + */ +public class MemoryMappedTxnStatusFileConfigTest { + + private static final String MEMORY_MAPPED_ENABLED_PROP = "org.eclipse.rdf4j.sail.nativerdf.MemoryMappedTxnStatusFile.enabled"; + + @TempDir + File dataDir; + + @AfterEach + public void clearProperty() { + System.clearProperty(MEMORY_MAPPED_ENABLED_PROP); + } + + @Test + public void defaultUsesNioTxnStatusFile() throws Exception { + TripleStore tripleStore = new TripleStore(dataDir, "spoc"); + try { + tripleStore.startTransaction(); + tripleStore.storeTriple(1, 2, 3, 4); + tripleStore.commit(); + } finally { + tripleStore.close(); + } + + File txnStatusFile = new File(dataDir, TxnStatusFile.FILE_NAME); + assertTrue(txnStatusFile.exists(), "Transaction status file should exist"); + assertEquals(0L, txnStatusFile.length(), + "Default TxnStatusFile implementation truncates the file for NONE status"); + } + + @Test + public void memoryMappedEnabledUsesFixedSizeFile() throws Exception { + System.setProperty(MEMORY_MAPPED_ENABLED_PROP, "true"); + + TripleStore tripleStore = new TripleStore(dataDir, "spoc"); + try { + tripleStore.startTransaction(); + tripleStore.storeTriple(1, 2, 3, 4); + tripleStore.commit(); + } finally { + tripleStore.close(); + } + + File txnStatusFile = new File(dataDir, TxnStatusFile.FILE_NAME); + assertTrue(txnStatusFile.exists(), "Transaction status file should exist"); + assertEquals(1L, txnStatusFile.length(), + "Memory-mapped TxnStatusFile keeps a single status byte on disk for NONE status"); + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/NativeOptimisticIsolationTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/NativeOptimisticIsolationTest.java index a3722842e51..7c4e2549518 100644 --- a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/NativeOptimisticIsolationTest.java +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/NativeOptimisticIsolationTest.java @@ -13,6 +13,8 @@ import org.eclipse.rdf4j.repository.config.RepositoryImplConfig; import org.eclipse.rdf4j.repository.sail.config.SailRepositoryConfig; import org.eclipse.rdf4j.repository.sail.config.SailRepositoryFactory; +import org.eclipse.rdf4j.sail.config.SailImplConfig; +import org.eclipse.rdf4j.sail.nativerdf.config.NativeStoreConfig; import org.eclipse.rdf4j.sail.nativerdf.config.NativeStoreFactory; import org.eclipse.rdf4j.testsuite.repository.OptimisticIsolationTest; import org.junit.AfterClass; @@ -25,7 +27,9 @@ public static void setUpClass() throws Exception { setRepositoryFactory(new SailRepositoryFactory() { @Override public RepositoryImplConfig getConfig() { - return new SailRepositoryConfig(new NativeStoreFactory().getConfig()); + NativeStoreConfig config = (NativeStoreConfig) new NativeStoreFactory().getConfig(); + config.setWalEnabled(false); + return new SailRepositoryConfig(config); } }); } diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/NativeSailStoreCorruptionTestIT.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/NativeSailStoreCorruptionTestIT.java index 2f75229837e..7ae6205c0a2 100644 --- a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/NativeSailStoreCorruptionTestIT.java +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/NativeSailStoreCorruptionTestIT.java @@ -29,13 +29,13 @@ import org.eclipse.rdf4j.model.util.Values; import org.eclipse.rdf4j.model.vocabulary.RDF; import org.eclipse.rdf4j.model.vocabulary.RDFS; -import org.eclipse.rdf4j.repository.Repository; import org.eclipse.rdf4j.repository.RepositoryConnection; import org.eclipse.rdf4j.repository.RepositoryResult; import org.eclipse.rdf4j.repository.sail.SailRepository; import org.eclipse.rdf4j.rio.RDFFormat; import org.eclipse.rdf4j.rio.RDFWriter; import org.eclipse.rdf4j.rio.Rio; +import org.eclipse.rdf4j.sail.nativerdf.wal.ValueStoreWalConfig; import org.jetbrains.annotations.NotNull; import org.junit.jupiter.api.AfterEach; import org.junit.jupiter.api.BeforeEach; @@ -58,7 +58,7 @@ public class NativeSailStoreCorruptionTestIT { @TempDir File tempFolder; - protected Repository repo; + protected SailRepository repo; protected final ValueFactory F = SimpleValueFactory.getInstance(); @@ -68,7 +68,11 @@ public class NativeSailStoreCorruptionTestIT { public void before() throws IOException { this.dataDir = new File(tempFolder, "dbmodel"); dataDir.mkdir(); - repo = new SailRepository(new NativeStore(dataDir, "spoc,posc")); + NativeStore sail = new NativeStore(dataDir, "spoc,posc"); + sail.setWalSyncPolicy(ValueStoreWalConfig.SyncPolicy.COMMIT); + sail.setWalEnabled(true); + repo = new SailRepository(sail); + repo.init(); IRI CTX_1 = F.createIRI("urn:one"); @@ -105,6 +109,11 @@ public void before() throws IOException { } + @AfterEach + public void tearDown() throws IOException { + NativeStore.SOFT_FAIL_ON_CORRUPT_DATA_AND_REPAIR_INDEXES = false; + } + public static void overwriteByteInFile(File valuesFile, long pos, int newVal) throws IOException { // Use RandomAccessFile in "rw" mode to read and write to the file @@ -151,7 +160,9 @@ public static void restoreFile(File dataDir, String s) throws IOException { } @Test +// @Timeout(30) public void testCorruptValuesDatFileNamespace() throws IOException { + String expected = getStatements().stream().map(Object::toString).reduce((a, b) -> a + "\n" + b).get(); repo.shutDown(); overwriteByteInFile(new File(dataDir, "values.dat"), 12, 0x0); @@ -160,10 +171,14 @@ public void testCorruptValuesDatFileNamespace() throws IOException { List list = getStatements(); assertEquals(6, list.size()); + String actual = list.stream().map(Object::toString).reduce((a, b) -> a + "\n" + b).get(); + assertEquals(expected, actual); } @Test +// @Timeout(30) public void testCorruptValuesDatFileNamespaceDatatype() throws IOException { + String expected = getStatements().stream().map(Object::toString).reduce((a, b) -> a + "\n" + b).get(); repo.shutDown(); overwriteByteInFile(new File(dataDir, "values.dat"), 96, 0x0); @@ -172,10 +187,14 @@ public void testCorruptValuesDatFileNamespaceDatatype() throws IOException { List list = getStatements(); assertEquals(6, list.size()); + String actual = list.stream().map(Object::toString).reduce((a, b) -> a + "\n" + b).get(); + assertEquals(expected, actual); } @Test +// @Timeout(30) public void testCorruptValuesDatFileEmptyDataArrayError() throws IOException { + String expected = getStatements().stream().map(Object::toString).reduce((a, b) -> a + "\n" + b).get(); repo.shutDown(); overwriteByteInFile(new File(dataDir, "values.dat"), 173, 0x0); @@ -184,10 +203,14 @@ public void testCorruptValuesDatFileEmptyDataArrayError() throws IOException { List list = getStatements(); assertEquals(6, list.size()); + String actual = list.stream().map(Object::toString).reduce((a, b) -> a + "\n" + b).get(); + assertEquals(expected, actual); } @Test +// @Timeout(30) public void testCorruptValuesDatFileInvalidTypeError() throws IOException { + String expected = getStatements().stream().map(Object::toString).reduce((a, b) -> a + "\n" + b).get(); repo.shutDown(); overwriteByteInFile(new File(dataDir, "values.dat"), 174, 0x0); @@ -196,10 +219,14 @@ public void testCorruptValuesDatFileInvalidTypeError() throws IOException { List list = getStatements(); assertEquals(6, list.size()); + String actual = list.stream().map(Object::toString).reduce((a, b) -> a + "\n" + b).get(); + assertEquals(expected, actual); } @Test +// @Timeout(30) public void testCorruptValuesDatFileEntireValuesDatFile() throws IOException { + String expected = getStatements().stream().map(Object::toString).reduce((a, b) -> a + "\n" + b).get(); for (int i = 4; i < 437; i++) { logger.debug("Corrupting byte at position " + i); repo.shutDown(); @@ -210,12 +237,16 @@ public void testCorruptValuesDatFileEntireValuesDatFile() throws IOException { repo.init(); List list = getStatements(); - assertEquals(6, list.size()); + assertEquals(6, list.size(), "Failed at byte position " + i); + String actual = list.stream().map(Object::toString).reduce((a, b) -> a + "\n" + b).get(); + assertEquals(expected, actual, "Failed at byte position " + i); } } @Test +// @Timeout(30) public void testCorruptLastByteOfValuesDatFile() throws IOException { + String expected = getStatements().stream().map(Object::toString).reduce((a, b) -> a + "\n" + b).get(); repo.shutDown(); File valuesFile = new File(dataDir, "values.dat"); long fileSize = valuesFile.length(); @@ -226,13 +257,17 @@ public void testCorruptLastByteOfValuesDatFile() throws IOException { List list = getStatements(); assertEquals(6, list.size()); + String actual = list.stream().map(Object::toString).reduce((a, b) -> a + "\n" + b).get(); + assertEquals(expected, actual); } @Test +// @Timeout(30) public void testCorruptValuesIdFile() throws IOException { repo.shutDown(); File valuesIdFile = new File(dataDir, "values.id"); long fileSize = valuesIdFile.length(); + String expected = getStatements().stream().map(Object::toString).reduce((a, b) -> a + "\n" + b).get(); for (long i = 4; i < fileSize; i++) { restoreFile(dataDir, "values.id"); @@ -240,29 +275,45 @@ public void testCorruptValuesIdFile() throws IOException { repo.init(); List list = getStatements(); assertEquals(6, list.size(), "Failed at byte position " + i); + String actual = getStatements().stream().map(Object::toString).reduce((a, b) -> a + "\n" + b).get(); + assertEquals(expected, actual, "Failed at byte position " + i); repo.shutDown(); } } @Test +// @Timeout(30) public void testCorruptValuesHashFile() throws IOException { repo.shutDown(); + + NativeStore sail = (NativeStore) repo.getSail(); + sail.setWalEnabled(false); + String file = "values.hash"; File nativeStoreFile = new File(dataDir, file); long fileSize = nativeStoreFile.length(); + String expected = getStatements().stream().map(Object::toString).reduce((a, b) -> a + "\n" + b).get(); for (long i = 4; i < fileSize; i++) { + if (i % 1024 == 0) { + System.out.println("Testing byte " + i); + } restoreFile(dataDir, file); overwriteByteInFile(nativeStoreFile, i, 0x0); repo.init(); List list = getStatements(); assertEquals(6, list.size(), "Failed at byte position " + i); + String actual = getStatements().stream().map(Object::toString).reduce((a, b) -> a + "\n" + b).get(); + assertEquals(expected, actual, "Failed at hash position " + i); + repo.shutDown(); } } @Test +// @Timeout(30) public void testCorruptValuesNamespacesFile() throws IOException { + String expected = getStatements().stream().map(Object::toString).reduce((a, b) -> a + "\n" + b).get(); repo.shutDown(); String file = "namespaces.dat"; File nativeStoreFile = new File(dataDir, file); @@ -274,12 +325,16 @@ public void testCorruptValuesNamespacesFile() throws IOException { repo.init(); List list = getStatements(); assertEquals(6, list.size(), "Failed at byte position " + i); + String actual = list.stream().map(Object::toString).reduce((a, b) -> a + "\n" + b).get(); + assertEquals(expected, actual, "Failed at byte position " + i); repo.shutDown(); } } @Test +// @Timeout(30) public void testCorruptValuesContextsFile() throws IOException { + String expected = getStatements().stream().map(Object::toString).reduce((a, b) -> a + "\n" + b).get(); repo.shutDown(); String file = "contexts.dat"; File nativeStoreFile = new File(dataDir, file); @@ -291,13 +346,19 @@ public void testCorruptValuesContextsFile() throws IOException { repo.init(); List list = getStatements(); assertEquals(6, list.size(), "Failed at byte position " + i); + String actual = list.stream().map(Object::toString).reduce((a, b) -> a + "\n" + b).get(); + assertEquals(expected, actual, "Failed at byte position " + i); repo.shutDown(); } } @Test +// @Timeout(30) public void testCorruptValuesPoscAllocFile() throws IOException { + String expected = getStatements().stream().map(Object::toString).reduce((a, b) -> a + "\n" + b).get(); repo.shutDown(); + ((NativeStore) repo.getSail()).setWalEnabled(false); + String file = "triples-posc.alloc"; File nativeStoreFile = new File(dataDir, file); long fileSize = nativeStoreFile.length(); @@ -306,67 +367,105 @@ public void testCorruptValuesPoscAllocFile() throws IOException { restoreFile(dataDir, file); overwriteByteInFile(nativeStoreFile, i, 0x0); repo.init(); + List list = getStatements(); assertEquals(6, list.size(), "Failed at byte position " + i); + String actual = list.stream().map(Object::toString).reduce((a, b) -> a + "\n" + b).get(); + assertEquals(expected, actual, "Failed at byte position " + i); repo.shutDown(); } } @Test +// @Timeout(30) public void testCorruptValuesPoscDataFile() throws IOException { + String expected = getStatements().stream().map(Object::toString).reduce((a, b) -> a + "\n" + b).get(); repo.shutDown(); + + ((NativeStore) repo.getSail()).setWalEnabled(false); + String file = "triples-posc.dat"; File nativeStoreFile = new File(dataDir, file); long fileSize = nativeStoreFile.length(); for (long i = 4; i < fileSize; i++) { + if (i % 1024 == 0) { + System.out.println("Testing byte " + i); + } NativeStore.SOFT_FAIL_ON_CORRUPT_DATA_AND_REPAIR_INDEXES = true; restoreFile(dataDir, file); overwriteByteInFile(nativeStoreFile, i, 0x0); + repo.init(); + List list = getStatements(); assertEquals(6, list.size(), "Failed at byte position " + i); + String actual = list.stream().map(Object::toString).reduce((a, b) -> a + "\n" + b).get(); + assertEquals(expected, actual, "Failed at byte position " + i); repo.shutDown(); } } @Test +// @Timeout(30) public void testCorruptValuesSpocAllocFile() throws IOException { + String expected = getStatements().stream().map(Object::toString).reduce((a, b) -> a + "\n" + b).get(); repo.shutDown(); + ((NativeStore) repo.getSail()).setWalEnabled(false); + String file = "triples-spoc.alloc"; File nativeStoreFile = new File(dataDir, file); long fileSize = nativeStoreFile.length(); for (long i = 4; i < fileSize; i++) { + if (i % 1024 == 0) { + System.out.println("Testing byte " + i); + } restoreFile(dataDir, file); overwriteByteInFile(nativeStoreFile, i, 0x0); repo.init(); List list = getStatements(); assertEquals(6, list.size(), "Failed at byte position " + i); + String actual = list.stream().map(Object::toString).reduce((a, b) -> a + "\n" + b).get(); + assertEquals(expected, actual, "Failed at byte position " + i); repo.shutDown(); } } @Test +// @Timeout(30) public void testCorruptValuesSpocDataFile() throws IOException { + String expected = getStatements().stream().map(Object::toString).reduce((a, b) -> a + "\n" + b).get(); repo.shutDown(); + ((NativeStore) repo.getSail()).setWalEnabled(false); + + NativeStore sail = (NativeStore) repo.getSail(); + sail.setWalEnabled(false); + String file = "triples-spoc.dat"; File nativeStoreFile = new File(dataDir, file); long fileSize = nativeStoreFile.length(); for (long i = 4; i < fileSize; i++) { + if (i % 1024 == 0) { + System.out.println("Testing byte " + i); + } restoreFile(dataDir, file); overwriteByteInFile(nativeStoreFile, i, 0x0); repo.init(); try { List list = getStatements(); assertEquals(6, list.size(), "Failed at byte position " + i); + String actual = list.stream().map(Object::toString).reduce((a, b) -> a + "\n" + b).get(); + assertEquals(expected, actual, "Failed at byte position " + i); } catch (Throwable ignored) { repo.shutDown(); nativeStoreFile.delete(); repo.init(); List list = getStatements(); assertEquals(6, list.size(), "Failed at byte position " + i); + String actual = list.stream().map(Object::toString).reduce((a, b) -> a + "\n" + b).get(); + assertEquals(expected, actual, "Failed at byte position " + i); } repo.shutDown(); diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/NativeSailStoreWalBootstrapTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/NativeSailStoreWalBootstrapTest.java new file mode 100644 index 00000000000..9ecf5d82c14 --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/NativeSailStoreWalBootstrapTest.java @@ -0,0 +1,59 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ + +package org.eclipse.rdf4j.sail.nativerdf; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.List; +import java.util.stream.Collectors; + +import org.eclipse.rdf4j.model.impl.SimpleValueFactory; +import org.eclipse.rdf4j.sail.nativerdf.wal.ValueStoreWalConfig; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +class NativeSailStoreWalBootstrapTest { + + @TempDir + Path tempDir; + + @Test + void enablingWalOnNonEmptyValueStoreRebuildsWal() throws Exception { + try (ValueStore store = new ValueStore(tempDir.toFile(), false)) { + store.storeValue(SimpleValueFactory.getInstance().createIRI("http://example.com/existing")); + } + + NativeStore nativeStore = new NativeStore(tempDir.toFile()); + try { + nativeStore.init(); + } finally { + nativeStore.shutDown(); + } + + Path walDir = tempDir.resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); +// Path marker = walDir.resolve("bootstrap.info"); +// assertThat(Files.exists(marker)).isTrue(); +// String markerContent = Files.readString(marker, StandardCharsets.UTF_8); +// assertThat(markerContent).contains("enabled-rebuild-existing-values"); + try (var stream = Files.list(walDir)) { + List segments = stream + .filter(p -> p.getFileName().toString().startsWith("wal-")) + .map(p -> p.getFileName().toString()) + .collect(Collectors.toList()); + assertThat(segments).isNotEmpty(); + assertThat(segments).allMatch(name -> name.matches("wal-[1-9]\\d*\\.v1(?:\\.gz)?")); + } + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/NativeStoreConcurrentValueStoreCorruptionTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/NativeStoreConcurrentValueStoreCorruptionTest.java index 731cd3911a1..71aa349e5bd 100644 --- a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/NativeStoreConcurrentValueStoreCorruptionTest.java +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/NativeStoreConcurrentValueStoreCorruptionTest.java @@ -21,6 +21,7 @@ import java.util.concurrent.Future; import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.io.TempDir; import org.junit.jupiter.api.parallel.Isolated; @@ -37,8 +38,8 @@ public class NativeStoreConcurrentValueStoreCorruptionTest { File dataDir; @AfterEach - public void resetSoftFailFlag() { - NativeStore.SOFT_FAIL_ON_CORRUPT_DATA_AND_REPAIR_INDEXES = true; + public void tearDown() { + NativeStore.SOFT_FAIL_ON_CORRUPT_DATA_AND_REPAIR_INDEXES = false; } @Test diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/NativeStoreConnectionTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/NativeStoreConnectionTest.java index 98bb8ebd657..796e192b37c 100644 --- a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/NativeStoreConnectionTest.java +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/NativeStoreConnectionTest.java @@ -27,7 +27,9 @@ public class NativeStoreConnectionTest extends RepositoryConnectionTest { @Override protected Repository createRepository(File dataDir) { - return new SailRepository(new NativeStore(dataDir, "spoc")); + NativeStore sail = new NativeStore(dataDir, "spoc"); + sail.setWalEnabled(false); + return new SailRepository(sail); } @ParameterizedTest diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/NativeStoreRepositoryCorruptionReproducerTestIT.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/NativeStoreRepositoryCorruptionReproducerTestIT.java index 33ca07ef208..0dc8bbd883b 100644 --- a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/NativeStoreRepositoryCorruptionReproducerTestIT.java +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/NativeStoreRepositoryCorruptionReproducerTestIT.java @@ -128,10 +128,9 @@ public void concurrentAddAndReadMayCorrupt() throws Exception { long until = System.nanoTime() + TimeUnit.SECONDS.toNanos(30); while (System.nanoTime() < until) { try (RepositoryResult statements = conn.getStatements(null, null, null, false)) { - statements.forEachRemaining(st -> { - st.toString(); - // no-op; force materialization - }); + // no-op; force materialization + // noinspection ResultOfMethodCallIgnored + statements.forEachRemaining(Object::toString); } Thread.onSpinWait(); } @@ -171,9 +170,8 @@ public void concurrentAddAndReadMayCorrupt() throws Exception { // If corruption occurred, iterating statements should throw RepositoryException try (RepositoryResult statements = conn.getStatements(null, null, null, false)) { - statements.forEachRemaining(st -> { - st.toString(); - }); + // noinspection ResultOfMethodCallIgnored + statements.forEachRemaining(Object::toString); } } diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/NativeStoreTxnTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/NativeStoreTxnTest.java index 83af7b715c0..3390903ead1 100644 --- a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/NativeStoreTxnTest.java +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/NativeStoreTxnTest.java @@ -20,6 +20,7 @@ import java.io.File; import java.nio.file.Files; +import java.util.Arrays; import org.eclipse.rdf4j.common.io.NioFile; import org.eclipse.rdf4j.model.IRI; @@ -33,6 +34,7 @@ import org.eclipse.rdf4j.repository.RepositoryConnection; import org.eclipse.rdf4j.repository.sail.SailRepository; import org.eclipse.rdf4j.sail.nativerdf.btree.RecordIterator; +import org.eclipse.rdf4j.sail.nativerdf.wal.ValueStoreWalConfig; import org.junit.jupiter.api.AfterEach; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; @@ -86,7 +88,11 @@ public void testTxncacheCleanup() throws Exception { for (File file : repoDir.listFiles()) { System.out.println("# " + file.getName()); } - assertEquals(15, repoDir.listFiles().length); + // With WAL enabled a 'wal' directory may be present; exclude it from the legacy count + int nonWalCount = (int) Arrays.stream(repoDir.listFiles()) + .filter(f -> !ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME.equals(f.getName())) + .count(); + assertEquals(15, nonWalCount); // make sure there is no txncacheXXX.dat file assertFalse(Files.list(repoDir.getAbsoluteFile().toPath()) diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/NativeStoreValueStoreCorruptionTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/NativeStoreValueStoreCorruptionTest.java index 60c6c580fa5..ed92561b8ca 100644 --- a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/NativeStoreValueStoreCorruptionTest.java +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/NativeStoreValueStoreCorruptionTest.java @@ -72,7 +72,9 @@ public void longLanguageTagShouldNotCorruptValueStore() throws Exception { @Test public void longLanguageTagShouldNotCorruptValueStoreIncremental() throws Exception { - SailRepository repo = new SailRepository(new NativeStore(dataDir)); + NativeStore sail = new NativeStore(dataDir); + sail.setWalEnabled(false); + SailRepository repo = new SailRepository(sail); repo.init(); for (int i = 0; i < 256; i++) { @@ -88,7 +90,9 @@ public void longLanguageTagShouldNotCorruptValueStoreIncremental() throws Except repo.shutDown(); - SailRepository reopened = new SailRepository(new NativeStore(dataDir)); + NativeStore sail1 = new NativeStore(dataDir); + sail1.setWalEnabled(false); + SailRepository reopened = new SailRepository(sail1); reopened.init(); try (RepositoryConnection connection = reopened.getConnection()) { diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/NativeStoreWalConfigTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/NativeStoreWalConfigTest.java new file mode 100644 index 00000000000..fc1aa99a6f5 --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/NativeStoreWalConfigTest.java @@ -0,0 +1,143 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.File; +import java.lang.reflect.Field; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.List; +import java.util.concurrent.ThreadLocalRandom; +import java.util.stream.Collectors; + +import org.eclipse.rdf4j.model.IRI; +import org.eclipse.rdf4j.model.impl.SimpleValueFactory; +import org.eclipse.rdf4j.repository.Repository; +import org.eclipse.rdf4j.repository.RepositoryConnection; +import org.eclipse.rdf4j.repository.sail.SailRepository; +import org.eclipse.rdf4j.sail.base.SailStore; +import org.eclipse.rdf4j.sail.base.SnapshotSailStore; +import org.eclipse.rdf4j.sail.nativerdf.config.NativeStoreConfig; +import org.eclipse.rdf4j.sail.nativerdf.config.NativeStoreFactory; +import org.eclipse.rdf4j.sail.nativerdf.wal.ValueStoreWAL; +import org.eclipse.rdf4j.sail.nativerdf.wal.ValueStoreWalConfig; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +class NativeStoreWalConfigTest { + + @TempDir + File dataDir; + + @Test + void respectsWalMaxSegmentBytes() throws Exception { + // Configure a very small WAL segment size to force rotation + NativeStoreConfig cfg = new NativeStoreConfig("spoc"); + cfg.setWalMaxSegmentBytes(32 * 1024); // 32 KiB + + NativeStoreFactory factory = new NativeStoreFactory(); + NativeStore sail = (NativeStore) factory.getSail(cfg); + sail.setDataDir(dataDir); + Repository repo = new SailRepository(sail); + repo.init(); + try (RepositoryConnection conn = repo.getConnection()) { + SimpleValueFactory vf = SimpleValueFactory.getInstance(); + IRI p = vf.createIRI("http://example.com/p"); + // Add enough statements with ~1KB literals to exceed 32 KiB + for (int i = 0; i < 200; i++) { + int len = 1024 + ThreadLocalRandom.current().nextInt(512); + String s = "x".repeat(len); + conn.add(vf.createIRI("http://example.com/s/" + i), p, vf.createLiteral(s)); + } + } + repo.shutDown(); + + // Verify multiple WAL segments were created due to small max size + Path walDir = dataDir.toPath().resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + assertThat(Files.isDirectory(walDir)).isTrue(); + try (var stream = Files.list(walDir)) { + List segments = stream + .filter(p -> p.getFileName().toString().matches("wal-[1-9]\\d*\\.v1(\\.gz)?")) + .collect(Collectors.toList()); + assertThat(segments.size()).as("expect >1 wal segments after forced rotation").isGreaterThan(1); + } + } + + @Test + void mapsAllWalConfigOptions() throws Exception { + NativeStoreConfig cfg = new NativeStoreConfig("spoc"); + cfg.setWalMaxSegmentBytes(1 << 20); // 1 MiB + cfg.setWalQueueCapacity(1234); + cfg.setWalBatchBufferBytes(1 << 14); // 16 KiB + cfg.setWalSyncPolicy("ALWAYS"); + cfg.setWalSyncIntervalMillis(50); + cfg.setWalIdlePollIntervalMillis(5); + cfg.setWalDirectoryName("custom-wal-dir"); + cfg.setWalSyncBootstrapOnOpen(true); + + NativeStoreFactory factory = new NativeStoreFactory(); + NativeStore sail = (NativeStore) factory.getSail(cfg); + sail.setDataDir(dataDir); + sail.init(); + + SailStore sailStore = sail.getSailStore(); + // unwrap SnapshotSailStore to get underlying NativeSailStore + Field backingField = SnapshotSailStore.class + .getDeclaredField("backingStore"); + backingField.setAccessible(true); + NativeSailStore nss = (NativeSailStore) backingField.get(sailStore); + + Field walField = NativeSailStore.class.getDeclaredField("valueStoreWal"); + walField.setAccessible(true); + ValueStoreWAL wal = (ValueStoreWAL) walField.get(nss); + ValueStoreWalConfig walCfg = wal.config(); + + assertThat(walCfg.maxSegmentBytes()).isEqualTo(1 << 20); + assertThat(walCfg.queueCapacity()).isEqualTo(1234); + assertThat(walCfg.batchBufferBytes()).isEqualTo(1 << 14); + assertThat(walCfg.syncPolicy()).isEqualTo(ValueStoreWalConfig.SyncPolicy.ALWAYS); + assertThat(walCfg.syncInterval().toMillis()).isEqualTo(50); + assertThat(walCfg.idlePollInterval().toMillis()).isEqualTo(5); + Path expectedWalDir = dataDir.toPath().resolve("custom-wal-dir"); + assertThat(walCfg.walDirectory()).isEqualTo(expectedWalDir); + assertThat(walCfg.snapshotsDirectory()).isEqualTo(expectedWalDir.resolve("snapshots")); + assertThat(walCfg.syncBootstrapOnOpen()).isTrue(); + } + + @Test + void disablesWalWhenConfigured() throws Exception { + NativeStoreConfig cfg = new NativeStoreConfig("spoc"); + cfg.setWalEnabled(false); + + NativeStoreFactory factory = new NativeStoreFactory(); + NativeStore sail = (NativeStore) factory.getSail(cfg); + sail.setDataDir(dataDir); + sail.init(); + try { + SailStore sailStore = sail.getSailStore(); + Field backingField = SnapshotSailStore.class.getDeclaredField("backingStore"); + backingField.setAccessible(true); + NativeSailStore nss = (NativeSailStore) backingField.get(sailStore); + + Field walField = NativeSailStore.class.getDeclaredField("valueStoreWal"); + walField.setAccessible(true); + Object wal = walField.get(nss); + assertThat(wal).as("WAL should be disabled when walEnabled=false").isNull(); + + Path walDir = dataDir.toPath().resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + assertThat(Files.exists(walDir)).isFalse(); + } finally { + sail.shutDown(); + } + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/QueryBenchmarkTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/QueryBenchmarkTest.java index 4ffb282d728..7680d88e8c8 100644 --- a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/QueryBenchmarkTest.java +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/QueryBenchmarkTest.java @@ -15,6 +15,7 @@ import java.io.IOException; import java.io.InputStream; import java.nio.charset.StandardCharsets; +import java.util.ArrayList; import java.util.List; import org.apache.commons.io.IOUtils; @@ -23,6 +24,7 @@ import org.eclipse.rdf4j.model.Resource; import org.eclipse.rdf4j.model.Statement; import org.eclipse.rdf4j.model.vocabulary.RDF; +import org.eclipse.rdf4j.repository.RepositoryResult; import org.eclipse.rdf4j.repository.sail.SailRepository; import org.eclipse.rdf4j.repository.sail.SailRepositoryConnection; import org.eclipse.rdf4j.rio.RDFFormat; @@ -60,22 +62,40 @@ public class QueryBenchmarkTest { @BeforeAll public static void beforeClass(@TempDir File dataDir) throws IOException { + System.out.println("Before class"); repository = new SailRepository(new NativeStore(dataDir, "spoc,ospc,psoc")); + System.out.println("Adding statements"); try (SailRepositoryConnection connection = repository.getConnection()) { connection.begin(IsolationLevels.NONE); connection.add(getResourceAsStream("benchmarkFiles/datagovbe-valid.ttl"), "", RDFFormat.TURTLE); connection.commit(); } + System.out.println("Getting statements"); + try (SailRepositoryConnection connection = repository.getConnection()) { - statementList = Iterations.asList(connection.getStatements(null, RDF.TYPE, null, false)); + try (RepositoryResult statements = connection.getStatements(null, RDF.TYPE, null, false)) { + statementList = new ArrayList<>(); + int i = 0; + while (statements.hasNext()) { + if (i++ % 10000 == 0) { + System.out.println("Loaded " + i + " statements"); + } + statementList.add(statements.next()); + } + } + } + System.out.println("GC"); + System.gc(); + System.out.println("Done"); + } private static InputStream getResourceAsStream(String name) { @@ -91,6 +111,7 @@ public static void afterClass() { @Test public void groupByQuery() { + System.out.println("groupByQuery"); try (SailRepositoryConnection connection = repository.getConnection()) { long count = connection .prepareTupleQuery(query1) @@ -103,6 +124,7 @@ public void groupByQuery() { @Test public void complexQuery() { + System.out.println("complexQuery"); try (SailRepositoryConnection connection = repository.getConnection()) { long count = connection .prepareTupleQuery(query4) @@ -115,6 +137,7 @@ public void complexQuery() { @Test public void distinctPredicatesQuery() { + System.out.println("distinctPredicatesQuery"); try (SailRepositoryConnection connection = repository.getConnection()) { long count = connection .prepareTupleQuery(query5) @@ -127,6 +150,7 @@ public void distinctPredicatesQuery() { @Test public void removeByQuery() { + System.out.println("removeByQuery"); try (SailRepositoryConnection connection = repository.getConnection()) { connection.begin(IsolationLevels.NONE); connection.remove((Resource) null, RDF.TYPE, null); @@ -141,6 +165,7 @@ public void removeByQuery() { @Test public void removeByQueryReadCommitted() { + System.out.println("removeByQueryReadCommitted"); try (SailRepositoryConnection connection = repository.getConnection()) { connection.begin(IsolationLevels.READ_COMMITTED); connection.remove((Resource) null, RDF.TYPE, null); @@ -155,6 +180,7 @@ public void removeByQueryReadCommitted() { @Test public void simpleUpdateQueryIsolationReadCommitted() { + System.out.println("simpleUpdateQueryIsolationReadCommitted"); try (SailRepositoryConnection connection = repository.getConnection()) { connection.begin(IsolationLevels.READ_COMMITTED); connection.prepareUpdate(query2).execute(); @@ -172,6 +198,7 @@ public void simpleUpdateQueryIsolationReadCommitted() { @Test public void simpleUpdateQueryIsolationNone() { + System.out.println("simpleUpdateQueryIsolationNone"); try (SailRepositoryConnection connection = repository.getConnection()) { connection.begin(IsolationLevels.NONE); diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/TripleStoreRecoveryTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/TripleStoreRecoveryTest.java index 18c6927e324..764fa44d719 100644 --- a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/TripleStoreRecoveryTest.java +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/TripleStoreRecoveryTest.java @@ -64,7 +64,7 @@ public void testCommitRecovery() throws Exception { // Pretend that commit was called TxnStatusFile txnStatusFile = new TxnStatusFile(dataDir); try { - txnStatusFile.setTxnStatus(TxnStatus.COMMITTING); + txnStatusFile.setTxnStatus(TxnStatus.COMMITTING, true); } finally { txnStatusFile.close(); } diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/ValueStoreRandomLookupTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/ValueStoreRandomLookupTest.java new file mode 100644 index 00000000000..004bde18ef5 --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/ValueStoreRandomLookupTest.java @@ -0,0 +1,365 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ + +package org.eclipse.rdf4j.sail.nativerdf; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.File; +import java.io.IOException; +import java.io.InputStream; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.Collections; +import java.util.Comparator; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.ThreadLocalRandom; +import java.util.regex.Matcher; +import java.util.regex.Pattern; +import java.util.stream.Collectors; +import java.util.zip.CRC32; +import java.util.zip.GZIPInputStream; + +import org.eclipse.rdf4j.common.transaction.IsolationLevels; +import org.eclipse.rdf4j.model.Value; +import org.eclipse.rdf4j.repository.sail.SailRepository; +import org.eclipse.rdf4j.repository.sail.SailRepositoryConnection; +import org.eclipse.rdf4j.rio.RDFFormat; +import org.eclipse.rdf4j.sail.SailException; +import org.eclipse.rdf4j.sail.nativerdf.config.NativeStoreConfig; +import org.eclipse.rdf4j.sail.nativerdf.config.NativeStoreFactory; +import org.eclipse.rdf4j.sail.nativerdf.datastore.DataStore; +import org.eclipse.rdf4j.sail.nativerdf.wal.ValueStoreWAL; +import org.eclipse.rdf4j.sail.nativerdf.wal.ValueStoreWalConfig; +import org.eclipse.rdf4j.sail.nativerdf.wal.ValueStoreWalReader; +import org.eclipse.rdf4j.sail.nativerdf.wal.ValueStoreWalRecord; +import org.eclipse.rdf4j.sail.nativerdf.wal.ValueStoreWalRecovery; +import org.eclipse.rdf4j.sail.nativerdf.wal.ValueStoreWalSearch; +import org.eclipse.rdf4j.sail.nativerdf.wal.ValueStoreWalTestUtils; +import org.eclipse.rdf4j.sail.nativerdf.wal.ValueStoreWalValueKind; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import com.fasterxml.jackson.core.JsonFactory; +import com.fasterxml.jackson.core.JsonParser; +import com.fasterxml.jackson.core.JsonToken; + +class ValueStoreRandomLookupTest { + + private static final Pattern SEGMENT_PATTERN = Pattern.compile("wal-(\\d+)\\.v1(?:\\.gz)?"); + private static final JsonFactory JSON_FACTORY = new JsonFactory(); + + @TempDir + File dataDir; + + @Test + void randomLookup50() throws Exception { + NativeStoreConfig cfg = new NativeStoreConfig("spoc,ospc,psoc"); + cfg.setWalMaxSegmentBytes(1024 * 1024 * 4); + NativeStore store = (NativeStore) new NativeStoreFactory().getSail(cfg); + store.setDataDir(dataDir); + SailRepository repository = new SailRepository(store); + repository.init(); + try (SailRepositoryConnection connection = repository.getConnection()) { + connection.begin(IsolationLevels.NONE); + try (InputStream in = getClass().getClassLoader() + .getResourceAsStream("benchmarkFiles/datagovbe-valid.ttl")) { + assertThat(in).as("benchmarkFiles/datagovbe-valid.ttl should be on classpath").isNotNull(); + connection.add(in, "", RDFFormat.TURTLE); + } + connection.commit(); + } + repository.shutDown(); + Path walDir = dataDir.toPath().resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + String storeUuid = Files.readString(walDir.resolve("store.uuid"), StandardCharsets.UTF_8).trim(); + + try (DataStore ds = new DataStore(dataDir, "values"); + ValueStore vs = new ValueStore(dataDir, false)) { + + int maxId = ds.getMaxID(); + assertThat(maxId).isGreaterThan(0); + + ValueStoreWalConfig walConfig = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid(storeUuid) + .build(); + Map statsBySegment = analyzeSegments(walDir, walConfig); + assertThat(statsBySegment).isNotEmpty(); + + ValueStoreWalRecovery recovery = new ValueStoreWalRecovery(); + Map dict; + try (ValueStoreWalReader reader = ValueStoreWalReader.open(walConfig)) { + dict = recovery.replay(reader); + } + assertThat(dict).isNotEmpty(); + + List ids = new ArrayList<>(); + for (Map.Entry entry : dict.entrySet()) { + ValueStoreWalValueKind kind = entry.getValue().valueKind(); + if (kind == ValueStoreWalValueKind.IRI || kind == ValueStoreWalValueKind.BNODE + || kind == ValueStoreWalValueKind.LITERAL) { + ids.add(entry.getKey()); + } + } + assertThat(ids).isNotEmpty(); + + List compressedStats = statsBySegment.values() + .stream() + .filter(SegmentStats::isCompressed) + .sorted(Comparator.comparingInt(SegmentStats::sequence)) + .collect(Collectors.toList()); + assertThat(compressedStats).isNotEmpty(); + for (SegmentStats stat : compressedStats) { + assertThat(stat.summaryLastId) + .as("Summary should exist for %s", stat.path.getFileName()) + .isNotNull(); + assertThat(stat.summaryCRC32) + .as("Summary CRC should exist for %s", stat.path.getFileName()) + .isNotNull(); + assertThat(stat.summaryLastId).isEqualTo(stat.highestMintedId); + long actualCrc = crc32(stat.uncompressedBytes, stat.summaryOffset); + assertThat(stat.summaryCRC32).isEqualTo(actualCrc); + } + + List orderedSegments = new ArrayList<>(statsBySegment.keySet()); + orderedSegments.sort(Comparator.comparingInt(p -> statsBySegment.get(p).sequence())); + assertThat(orderedSegments).isNotEmpty(); + Path firstSegment = orderedSegments.get(0); + Path currentSegment = orderedSegments.get(orderedSegments.size() - 1); + + Set deleted = new HashSet<>(); + Files.deleteIfExists(firstSegment); + deleted.add(firstSegment); + Files.deleteIfExists(currentSegment); + deleted.add(currentSegment); + + ThreadLocalRandom random = ThreadLocalRandom.current(); + for (Path segment : orderedSegments) { + if (deleted.contains(segment)) { + continue; + } + if (random.nextBoolean()) { + Files.deleteIfExists(segment); + deleted.add(segment); + } + } + + Set deletedIds = new HashSet<>(); + Set survivingIds = new HashSet<>(); + for (Map.Entry entry : statsBySegment.entrySet()) { + if (deleted.contains(entry.getKey())) { + deletedIds.addAll(entry.getValue().mintedIds); + } else { + survivingIds.addAll(entry.getValue().mintedIds); + } + } + + ValueStoreWalSearch search = ValueStoreWalSearch.open(walConfig); + int walMatches = 0; + for (int i = 0; i < 50; i++) { + int id = ids.get(random.nextInt(ids.size())); + assertThat(id).isBetween(1, maxId); + Value value = null; + try { + value = vs.getValue(id); + } catch (SailException e) { + if (!deletedIds.contains(id)) { + throw e; + } + } + Value walValue = search.findValueById(id); + if (deletedIds.contains(id)) { + assertThat(walValue).as("wal search should miss deleted segment id %s", id).isNull(); + continue; + } + assertThat(value).as("ValueStore value not null for surviving id %s", id).isNotNull(); + assertThat(walValue).as("wal search should recover surviving id %s", id).isEqualTo(value); + walMatches++; + } + assertThat(walMatches).as("should recover at least one id via WAL").isGreaterThan(0); + + List survivorList = new ArrayList<>(survivingIds); + Collections.shuffle(survivorList); + int sampleCount = Math.min(50, survivorList.size()); + for (int i = 0; i < sampleCount; i++) { + int id = survivorList.get(i); + Value expected = vs.getValue(id); + Value fromWal = search.findValueById(id); + assertThat(expected).isNotNull(); + assertThat(fromWal).isEqualTo(expected); + } + assertThat(sampleCount).as("should have surviving ids to verify").isGreaterThan(0); + + int found = 0; + for (int i = 0; i < 50; i++) { + int id = ids.get(random.nextInt(ids.size())); + assertThat(id).isBetween(1, maxId); + Value v = vs.getValue(id); + Value w = search.findValueById(id); + if (w != null && v != null && v.equals(w)) { + found++; + } + } + assertThat(found).as("Should resolve values for surviving WAL segments").isGreaterThan(0); + } + } + + private static Map analyzeSegments(Path walDir, ValueStoreWalConfig config) throws IOException { + Map stats = new HashMap<>(); + if (!Files.isDirectory(walDir)) { + return stats; + } + try (var stream = Files.list(walDir)) { + for (Path path : stream.collect(Collectors.toList())) { + Matcher matcher = SEGMENT_PATTERN.matcher(path.getFileName().toString()); + if (matcher.matches()) { + stats.put(path, analyzeSingleSegment(path)); + } + } + } + return stats; + } + + private static SegmentStats analyzeSingleSegment(Path path) throws IOException { + boolean compressed = path.getFileName().toString().endsWith(".gz"); + byte[] content; + if (compressed) { + try (GZIPInputStream gin = new GZIPInputStream(Files.newInputStream(path))) { + content = gin.readAllBytes(); + } + } else { + content = Files.readAllBytes(path); + } + int sequence = ValueStoreWalTestUtils.readSegmentSequence(content); + SegmentStats stats = new SegmentStats(path, sequence, compressed, content); + ByteBuffer buffer = ByteBuffer.wrap(content).order(ByteOrder.LITTLE_ENDIAN); + while (buffer.remaining() >= Integer.BYTES) { + int frameStart = buffer.position(); + int length = buffer.getInt(); + if (length <= 0 || length > ValueStoreWAL.MAX_FRAME_BYTES) { + break; + } + if (buffer.remaining() < length + Integer.BYTES) { + break; + } + byte[] json = new byte[length]; + buffer.get(json); + buffer.getInt(); + ParsedRecord record = ParsedRecord.parse(json); + if (record.type == 'M') { + if (record.kind == ValueStoreWalValueKind.IRI || record.kind == ValueStoreWalValueKind.BNODE + || record.kind == ValueStoreWalValueKind.LITERAL) { + stats.mintedIds.add(record.id); + } + stats.highestMintedId = Math.max(stats.highestMintedId, record.id); + } else if (record.type == 'S' && compressed) { + stats.summaryLastId = record.id; + stats.summaryCRC32 = record.crc32; + stats.summaryOffset = frameStart; + break; + } + } + return stats; + } + + private static long crc32(byte[] content, int limit) { + if (limit <= 0) { + return 0L; + } + CRC32 crc32 = new CRC32(); + crc32.update(content, 0, Math.min(limit, content.length)); + return crc32.getValue(); + } + + private static final class SegmentStats { + final Path path; + final int sequence; + final boolean compressed; + final byte[] uncompressedBytes; + final List mintedIds = new ArrayList<>(); + Integer summaryLastId; + Long summaryCRC32; + int summaryOffset = -1; + int highestMintedId = 0; + + SegmentStats(Path path, int sequence, boolean compressed, byte[] uncompressedBytes) { + this.path = path; + this.sequence = sequence; + this.compressed = compressed; + this.uncompressedBytes = uncompressedBytes; + } + + boolean isCompressed() { + return compressed; + } + + int sequence() { + return sequence; + } + } + + private static final class ParsedRecord { + final char type; + final int id; + final long crc32; + final ValueStoreWalValueKind kind; + final int segment; + + ParsedRecord(char type, int id, long crc32, ValueStoreWalValueKind kind, int segment) { + this.type = type; + this.id = id; + this.crc32 = crc32; + this.kind = kind; + this.segment = segment; + } + + static ParsedRecord parse(byte[] json) throws IOException { + try (JsonParser parser = JSON_FACTORY.createParser(json)) { + char type = '?'; + int id = 0; + long crc32 = 0L; + ValueStoreWalValueKind kind = ValueStoreWalValueKind.NAMESPACE; + int segment = 0; + while (parser.nextToken() != null) { + JsonToken token = parser.currentToken(); + if (token == JsonToken.FIELD_NAME) { + String field = parser.getCurrentName(); + parser.nextToken(); + if ("t".equals(field)) { + String value = parser.getValueAsString(""); + type = value.isEmpty() ? '?' : value.charAt(0); + } else if ("id".equals(field) || "lastId".equals(field)) { + id = parser.getValueAsInt(0); + } else if ("crc32".equals(field)) { + crc32 = parser.getValueAsLong(0L); + } else if ("vk".equals(field)) { + String code = parser.getValueAsString(""); + kind = ValueStoreWalValueKind.fromCode(code); + } else if ("segment".equals(field)) { + segment = parser.getValueAsInt(0); + } else { + parser.skipChildren(); + } + } + } + return new ParsedRecord(type, id, crc32, kind, segment); + } + } + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/benchmark/TransactionsPerSecondBenchmark.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/benchmark/TransactionsPerSecondBenchmark.java index 635e456b819..ddb199db1cf 100644 --- a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/benchmark/TransactionsPerSecondBenchmark.java +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/benchmark/TransactionsPerSecondBenchmark.java @@ -48,8 +48,8 @@ @State(Scope.Benchmark) @Warmup(iterations = 2) @BenchmarkMode({ Mode.Throughput }) -@Fork(value = 1, jvmArgs = { "-Xms8G", "-Xmx8G", "-XX:+UseG1GC" }) -//@Fork(value = 1, jvmArgs = {"-Xms8G", "-Xmx8G", "-XX:+UseG1GC", "-XX:+UnlockCommercialFeatures", "-XX:StartFlightRecording=delay=60s,duration=120s,filename=recording.jfr,settings=profile", "-XX:FlightRecorderOptions=samplethreads=true,stackdepth=1024", "-XX:+UnlockDiagnosticVMOptions", "-XX:+DebugNonSafepoints"}) +@Fork(value = 1, jvmArgs = { "-Xms4G", "-Xmx4G", "-XX:+UseG1GC" }) +//@Fork(value = 1, jvmArgs = {"-Xms4G", "-Xmx4G", "-XX:StartFlightRecording=jdk.CPUTimeSample#enabled=true,filename=profile.jfr,method-profiling=max","-XX:FlightRecorderOptions=stackdepth=1024", "-XX:+UnlockDiagnosticVMOptions", "-XX:+DebugNonSafepoints"}) @Measurement(iterations = 3) @OutputTimeUnit(TimeUnit.SECONDS) public class TransactionsPerSecondBenchmark { @@ -84,6 +84,7 @@ public void beforeClass() { NativeStore sail = new NativeStore(file, "spoc,ospc,psoc"); sail.setForceSync(forceSync); + sail.setWalIdlePollIntervalMillis(100); repository = new SailRepository(sail); connection = repository.getConnection(); random = new Random(1337); @@ -133,6 +134,15 @@ public void transactionsLevelNone() { connection.commit(); } + @Benchmark + public void mediumTransactionsLevelSnapshotRead() { + connection.begin(IsolationLevels.SNAPSHOT_READ); + for (int k = 0; k < 10; k++) { + connection.add(randomResource(), randomPredicate(), literalGenerator.createRandomLiteral()); + } + connection.commit(); + } + @Benchmark public void mediumTransactionsLevelNone() { connection.begin(IsolationLevels.NONE); diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/btree/BTreeTestRuns.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/btree/BTreeTestRuns.java index cde8842085f..76f273fb78f 100644 --- a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/btree/BTreeTestRuns.java +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/btree/BTreeTestRuns.java @@ -11,6 +11,7 @@ package org.eclipse.rdf4j.sail.nativerdf.btree; import java.io.File; +import java.util.Random; public class BTreeTestRuns { /*--------------* @@ -34,7 +35,7 @@ public static void runPerformanceTest(String[] args) throws Exception { RecordComparator comparator = new DefaultRecordComparator(); try (BTree btree = new BTree(dataDir, filenamePrefix, 501, 13, comparator)) { - java.util.Random random = new java.util.Random(0L); + Random random = new Random(0L); byte[] value = new byte[13]; long startTime = System.currentTimeMillis(); diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/btree/NodeListenerRegistryPerformanceTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/btree/NodeListenerRegistryPerformanceTest.java new file mode 100644 index 00000000000..27f0b508272 --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/btree/NodeListenerRegistryPerformanceTest.java @@ -0,0 +1,117 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.btree; + +import java.io.File; +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; +import java.util.concurrent.TimeUnit; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +class NodeListenerRegistryPerformanceTest { + + @Test + void deregistrationOfLargeListenerSetCompletesQuickly(@TempDir File dataDir) throws IOException { + try (BTree tree = new BTree(dataDir, "listener", 4096, 64)) { + Node node = new Node(1, tree); + int listenerCount = 120_000; + NodeListener[] listeners = new NodeListener[listenerCount]; + + for (int i = 0; i < listenerCount; i++) { + listeners[i] = new NoOpNodeListener(); + node.register(listeners[i]); + } + + long started = System.nanoTime(); + for (int i = listenerCount - 1; i >= 0; i--) { + node.deregister(listeners[i]); + } + long elapsedMillis = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - started); + + Assertions.assertTrue(elapsedMillis < 5_000, + () -> "deregistering " + listenerCount + " listeners took " + elapsedMillis + "ms"); + } + } + + @Test + void concurrentRegistrationsDoNotLeak(@TempDir File dataDir) throws Exception { + try (BTree tree = new BTree(dataDir, "listener-concurrent", 4096, 64)) { + Node node = new Node(2, tree); + int threads = Math.max(4, Runtime.getRuntime().availableProcessors()); + ExecutorService executor = Executors.newFixedThreadPool(threads); + CountDownLatch latch = new CountDownLatch(1); + List> futures = new ArrayList<>(); + for (int t = 0; t < threads; t++) { + futures.add(executor.submit(() -> { + latch.await(); + for (int i = 0; i < 5_000; i++) { + NodeListener listener = new NoOpNodeListener(); + NodeListenerHandle handle = node.register(listener); + if ((i & 1) == 0) { + handle.remove(); + } else { + node.deregister(listener); + } + } + return null; + })); + } + latch.countDown(); + for (Future future : futures) { + future.get(); + } + executor.shutdown(); + executor.awaitTermination(10, TimeUnit.SECONDS); + Assertions.assertEquals(0, node.getRegisteredListenerCount()); + } + } + + private static final class NoOpNodeListener implements NodeListener { + + @Override + public boolean valueAdded(Node node, int addedIndex) { + return false; + } + + @Override + public boolean valueRemoved(Node node, int removedIndex) { + return false; + } + + @Override + public boolean rotatedLeft(Node node, int valueIndex, Node leftChildNode, Node rightChildNode) { + return false; + } + + @Override + public boolean rotatedRight(Node node, int valueIndex, Node leftChildNode, Node rightChildNode) { + return false; + } + + @Override + public boolean nodeSplit(Node node, Node newNode, int medianIdx) { + return false; + } + + @Override + public boolean nodeMergedWith(Node sourceNode, Node targetNode, int mergeIdx) { + return false; + } + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/btree/NodeSearchTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/btree/NodeSearchTest.java new file mode 100644 index 00000000000..1f87a328d7e --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/btree/NodeSearchTest.java @@ -0,0 +1,63 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + ******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.btree; + +import static org.junit.jupiter.api.Assertions.assertEquals; + +import java.io.File; + +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +class NodeSearchTest { + + @TempDir + File tempDir; + + private BTree tree; + + @BeforeEach + void setUp() throws Exception { + tree = new BTree(tempDir, "node-search", 85, 1); + } + + @AfterEach + void tearDown() throws Exception { + if (tree != null) { + tree.delete(); + } + } + + @Test + void exactMatchesAndInsertionPoints() { + Node node = new Node(1, tree); + appendValue(node, 10); + appendValue(node, 20); + appendValue(node, 30); + appendValue(node, 40); + + assertEquals(0, node.search(bytes(10))); + assertEquals(3, node.search(bytes(40))); + assertEquals(-1, node.search(bytes(5))); + assertEquals(-3, node.search(bytes(25))); + assertEquals(-5, node.search(bytes(50))); + } + + private static void appendValue(Node node, int value) { + node.insertValueNodeIDPair(node.getValueCount(), bytes(value), 0); + } + + private static byte[] bytes(int value) { + return new byte[] { (byte) value }; + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/datastore/DataStoreRecoveryTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/datastore/DataStoreRecoveryTest.java index 9c2d7675796..9f47051d251 100644 --- a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/datastore/DataStoreRecoveryTest.java +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/datastore/DataStoreRecoveryTest.java @@ -32,17 +32,14 @@ public class DataStoreRecoveryTest { @TempDir File tempDir; - private boolean previousSoftFlag; - @BeforeEach public void setup() { - previousSoftFlag = NativeStore.SOFT_FAIL_ON_CORRUPT_DATA_AND_REPAIR_INDEXES; NativeStore.SOFT_FAIL_ON_CORRUPT_DATA_AND_REPAIR_INDEXES = true; } @AfterEach public void teardown() { - NativeStore.SOFT_FAIL_ON_CORRUPT_DATA_AND_REPAIR_INDEXES = previousSoftFlag; + NativeStore.SOFT_FAIL_ON_CORRUPT_DATA_AND_REPAIR_INDEXES = false; } @Test diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/testutil/FailureInjectingFileChannel.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/testutil/FailureInjectingFileChannel.java new file mode 100644 index 00000000000..2ad8e4bc988 --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/testutil/FailureInjectingFileChannel.java @@ -0,0 +1,146 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.testutil; + +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.MappedByteBuffer; +import java.nio.channels.FileChannel; +import java.nio.channels.FileLock; +import java.nio.channels.ReadableByteChannel; +import java.nio.channels.WritableByteChannel; + +/** + * Delegating FileChannel that can simulate failures for testing. + */ +public class FailureInjectingFileChannel extends FileChannel { + + private final FileChannel delegate; + + // simple toggles for simulation + private volatile boolean failNextWrite; + private volatile boolean failNextForce; + + public FailureInjectingFileChannel(FileChannel delegate) { + this.delegate = delegate; + } + + public void setFailNextWrite(boolean fail) { + this.failNextWrite = fail; + } + + public void setFailNextForce(boolean fail) { + this.failNextForce = fail; + } + + @Override + public int read(ByteBuffer dst) throws IOException { + return delegate.read(dst); + } + + @Override + public long read(ByteBuffer[] dsts, int offset, int length) throws IOException { + return delegate.read(dsts, offset, length); + } + + @Override + public int write(ByteBuffer src) throws IOException { + if (failNextWrite) { + failNextWrite = false; + throw new IOException("Simulated write failure"); + } + return delegate.write(src); + } + + @Override + public long write(ByteBuffer[] srcs, int offset, int length) throws IOException { + if (failNextWrite) { + failNextWrite = false; + throw new IOException("Simulated write failure"); + } + return delegate.write(srcs, offset, length); + } + + @Override + public long position() throws IOException { + return delegate.position(); + } + + @Override + public FileChannel position(long newPosition) throws IOException { + delegate.position(newPosition); + return this; + } + + @Override + public long size() throws IOException { + return delegate.size(); + } + + @Override + public FileChannel truncate(long size) throws IOException { + delegate.truncate(size); + return this; + } + + @Override + public void force(boolean metaData) throws IOException { + if (failNextForce) { + failNextForce = false; + throw new IOException("Simulated force failure"); + } + delegate.force(metaData); + } + + @Override + public long transferTo(long position, long count, WritableByteChannel target) throws IOException { + return delegate.transferTo(position, count, target); + } + + @Override + public long transferFrom(ReadableByteChannel src, long position, long count) throws IOException { + return delegate.transferFrom(src, position, count); + } + + @Override + public int read(ByteBuffer dst, long position) throws IOException { + return delegate.read(dst, position); + } + + @Override + public int write(ByteBuffer src, long position) throws IOException { + if (failNextWrite) { + failNextWrite = false; + throw new IOException("Simulated write failure"); + } + return delegate.write(src, position); + } + + @Override + protected void implCloseChannel() throws IOException { + delegate.close(); + } + + @Override + public FileLock lock(long position, long size, boolean shared) throws IOException { + return delegate.lock(position, size, shared); + } + + @Override + public FileLock tryLock(long position, long size, boolean shared) throws IOException { + return delegate.tryLock(position, size, shared); + } + + @Override + public MappedByteBuffer map(MapMode mode, long position, long size) throws IOException { + return delegate.map(mode, position, size); + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/JmhRunnerHarness.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/JmhRunnerHarness.java new file mode 100644 index 00000000000..9c5cf443d37 --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/JmhRunnerHarness.java @@ -0,0 +1,80 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ + +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import org.openjdk.jmh.annotations.Mode; +import org.openjdk.jmh.results.format.ResultFormatType; +import org.openjdk.jmh.runner.Runner; +import org.openjdk.jmh.runner.options.Options; +import org.openjdk.jmh.runner.options.OptionsBuilder; +import org.openjdk.jmh.runner.options.TimeValue; + +/** + * Simple harness to run JMH benchmarks from the IDE or via a Java main. + * + * System properties (optional): -Djmh.include=regex (default: ".*Wal.*Benchmark.*") -Djmh.threads=N (default: 8) + * -Djmh.forks=N (default: 1) -Djmh.warmupIterations=N (default: 3) -Djmh.measurementIterations=N (default: 5) + * -Djmh.warmupTimeSeconds=N (default: 2) -Djmh.measurementTimeSeconds=N (default: 3) + * -Djmh.mode=THROUGHPUT|SAMPLE_TIME|... (default: THROUGHPUT) -Djmh.result=path (optional) + * -Djmh.result.format=text|json|csv (default: text if result provided) + */ +public final class JmhRunnerHarness { + + private JmhRunnerHarness() { + } + + public static void main(String[] args) throws Exception { + String include = System.getProperty("jmh.include", ".*Wal.*Benchmark.*"); + int threads = Integer.getInteger("jmh.threads", 8); + int forks = Integer.getInteger("jmh.forks", 1); + int warmupIterations = Integer.getInteger("jmh.warmupIterations", 3); + int measurementIterations = Integer.getInteger("jmh.measurementIterations", 5); + int warmupTimeSec = Integer.getInteger("jmh.warmupTimeSeconds", 2); + int measurementTimeSec = Integer.getInteger("jmh.measurementTimeSeconds", 3); + String modeProp = System.getProperty("jmh.mode", "THROUGHPUT").toUpperCase(); + + OptionsBuilder builder = new OptionsBuilder(); + builder.include(include) + .threads(threads) + .forks(forks) + .warmupIterations(warmupIterations) + .measurementIterations(measurementIterations) + .warmupTime(TimeValue.seconds(warmupTimeSec)) + .measurementTime(TimeValue.seconds(measurementTimeSec)); + + try { + builder.mode(Mode.valueOf(modeProp)); + } catch (IllegalArgumentException ignored) { + builder.mode(Mode.Throughput); + } + + String resultPath = System.getProperty("jmh.result", "").trim(); + if (!resultPath.isEmpty()) { + String fmt = System.getProperty("jmh.result.format", "text").toLowerCase(); + ResultFormatType rft = ResultFormatType.TEXT; + switch (fmt) { + case "json": + rft = ResultFormatType.JSON; + break; + case "csv": + rft = ResultFormatType.CSV; + break; + default: + rft = ResultFormatType.TEXT; + } + builder.result(resultPath).resultFormat(rft); + } + + Options options = builder.build(); + new Runner(options).run(); + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWALForceOnRotateTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWALForceOnRotateTest.java new file mode 100644 index 00000000000..9954cf29b95 --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWALForceOnRotateTest.java @@ -0,0 +1,213 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ + +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.IOException; +import java.lang.reflect.Field; +import java.lang.reflect.Method; +import java.nio.ByteBuffer; +import java.nio.MappedByteBuffer; +import java.nio.channels.FileChannel; +import java.nio.channels.FileLock; +import java.nio.channels.ReadableByteChannel; +import java.nio.channels.WritableByteChannel; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.UUID; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicLong; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +/** + * Proves that rotating a WAL segment must force the previous segment to disk before closing it. This test wraps the + * writer's FileChannel with a tracking wrapper and invokes the private rotate method via reflection. + */ +class ValueStoreWALForceOnRotateTest { + + @TempDir + Path tempDir; + + @Test + void rotationForcesPreviousSegment() throws Exception { + Path walDir = tempDir.resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + Files.createDirectories(walDir); + + ValueStoreWalConfig cfg = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .syncPolicy(ValueStoreWalConfig.SyncPolicy.COMMIT) + .storeUuid(UUID.randomUUID().toString()) + .maxSegmentBytes(1 << 20) // 1MB; size irrelevant since we call rotate directly + .build(); + + try (ValueStoreWAL wal = ValueStoreWAL.open(cfg)) { + // Mint a single record to ensure lastAppendedLsn > lastForcedLsn so a force would be required. + long lsn = wal.logMint(1, ValueStoreWalValueKind.LITERAL, "x", "http://dt", "", 123); + + // Wait until the writer thread has actually appended the record (no force requested!) + waitUntilLastAppendedAtLeast(wal, lsn); + + Object logWriter = getField(wal, "logWriter"); + + // Wrap the current segment channel with a tracker and swap it in. + FileChannel original = (FileChannel) getField(logWriter, "segmentChannel"); + TrackingFileChannel tracking = new TrackingFileChannel(original); + setField(logWriter, "segmentChannel", tracking); + + // Call the private rotate method directly so we only exercise rotation (no additional forces). + Method rotate = logWriter.getClass().getDeclaredMethod("rotateSegment"); + rotate.setAccessible(true); + rotate.invoke(logWriter); + + // Expectation: rotation must force() the old segment before closing it + assertThat(tracking.wasForced()).as("previous segment must be fsynced before rotation").isTrue(); + } + } + + private static void waitUntilLastAppendedAtLeast(ValueStoreWAL wal, long targetLsn) throws Exception { + Field f = ValueStoreWAL.class.getDeclaredField("lastAppendedLsn"); + f.setAccessible(true); + AtomicLong lastAppended = (AtomicLong) f.get(wal); + long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(5); + while (System.nanoTime() < deadline) { + if (lastAppended.get() >= targetLsn) { + return; + } + Thread.sleep(1); + } + throw new AssertionError("writer thread did not append record in time"); + } + + /** + * Minimal FileChannel wrapper that tracks whether force() was called, delegating all operations to the wrapped + * channel. + */ + private static final class TrackingFileChannel extends FileChannel { + private final FileChannel delegate; + private volatile boolean forced; + + TrackingFileChannel(FileChannel delegate) { + this.delegate = delegate; + } + + boolean wasForced() { + return forced; + } + + @Override + public void force(boolean metaData) throws IOException { + forced = true; + delegate.force(metaData); + } + + // --- Delegate all abstract methods --- + + @Override + public int read(ByteBuffer dst) throws IOException { + return delegate.read(dst); + } + + @Override + public long read(ByteBuffer[] dsts, int offset, int length) throws IOException { + return delegate.read(dsts, offset, length); + } + + @Override + public int write(ByteBuffer src) throws IOException { + return delegate.write(src); + } + + @Override + public long write(ByteBuffer[] srcs, int offset, int length) throws IOException { + return delegate.write(srcs, offset, length); + } + + @Override + public long position() throws IOException { + return delegate.position(); + } + + @Override + public FileChannel position(long newPosition) throws IOException { + delegate.position(newPosition); + return this; + } + + @Override + public long size() throws IOException { + return delegate.size(); + } + + @Override + public FileChannel truncate(long size) throws IOException { + delegate.truncate(size); + return this; + } + + @Override + public long transferTo(long position, long count, WritableByteChannel target) + throws IOException { + return delegate.transferTo(position, count, target); + } + + @Override + public long transferFrom(ReadableByteChannel src, long position, long count) + throws IOException { + return delegate.transferFrom(src, position, count); + } + + @Override + public MappedByteBuffer map(MapMode mode, long position, long size) throws IOException { + return delegate.map(mode, position, size); + } + + @Override + public int read(ByteBuffer dst, long position) throws IOException { + return delegate.read(dst, position); + } + + @Override + public int write(ByteBuffer src, long position) throws IOException { + return delegate.write(src, position); + } + + @Override + protected void implCloseChannel() throws IOException { + delegate.close(); + } + + @Override + public FileLock lock(long position, long size, boolean shared) throws IOException { + return delegate.lock(position, size, shared); + } + + @Override + public FileLock tryLock(long position, long size, boolean shared) throws IOException { + return delegate.tryLock(position, size, shared); + } + } + + private static Object getField(Object target, String name) throws Exception { + Field f = target.getClass().getDeclaredField(name); + f.setAccessible(true); + return f.get(target); + } + + private static void setField(Object target, String name, Object value) throws Exception { + Field f = target.getClass().getDeclaredField(name); + f.setAccessible(true); + f.set(target, value); + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWALGzipSafetyTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWALGzipSafetyTest.java new file mode 100644 index 00000000000..04ffcde53bf --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWALGzipSafetyTest.java @@ -0,0 +1,74 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ + +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.UUID; +import java.util.zip.GZIPInputStream; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +/** + * Tests gzip safety: we don't delete the original segment if compression fails, and resulting gzip fully decompresses. + */ +class ValueStoreWALGzipSafetyTest { + + @TempDir + Path tempDir; + + @Test + void gzipContainsFullData() throws Exception { + Path walDir = tempDir.resolve("wal2"); + Files.createDirectories(walDir); + ValueStoreWalConfig cfg = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid(UUID.randomUUID().toString()) + .maxSegmentBytes(4096) + .build(); + // Generate enough data to force at least one gzip segment + long lastLsn; + try (ValueStoreWAL wal = ValueStoreWAL.open(cfg)) { + lastLsn = -1; + for (int i = 0; i < 500; i++) { + lastLsn = wal.logMint(i + 1, ValueStoreWalValueKind.LITERAL, "v" + i, "http://dt", "", i * 31); + } + wal.awaitDurable(lastLsn); + } + + // Find a gzip segment and fully decompress it, asserting we reach EOF and read > 0 bytes + Path gz = Files.list(walDir) + .filter(p -> p.getFileName().toString().endsWith(".v1.gz")) + .findFirst() + .orElseThrow(() -> new IOException("no gzip segment found")); + + long total = 0; + byte[] buf = new byte[1 << 15]; + try (GZIPInputStream in = new GZIPInputStream(Files.newInputStream(gz))) { + int r; + while ((r = in.read(buf)) >= 0) { + total += r; + } + } + assertThat(total).isGreaterThan(0L); + } + + private static Object getField(Object target, String name) throws Exception { + var f = target.getClass().getDeclaredField(name); + f.setAccessible(true); + return f.get(target); + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWALMonotonicLsnTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWALMonotonicLsnTest.java new file mode 100644 index 00000000000..966f38ebea9 --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWALMonotonicLsnTest.java @@ -0,0 +1,50 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.File; +import java.nio.file.Path; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +class ValueStoreWALMonotonicLsnTest { + + @TempDir + File tempDir; + + @Test + void lsnMonotonicAcrossRestart() throws Exception { + Path walDir = tempDir.toPath().resolve("wal"); + + ValueStoreWalConfig config = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid("test-store-uuid") + .build(); + + long firstLsn; + try (ValueStoreWAL wal = ValueStoreWAL.open(config)) { + firstLsn = wal.logMint(1, ValueStoreWalValueKind.IRI, "lex", "dt", "en", 123); + wal.awaitDurable(firstLsn); + } + + long secondLsn; + try (ValueStoreWAL wal = ValueStoreWAL.open(config)) { + secondLsn = wal.logMint(2, ValueStoreWalValueKind.IRI, "lex2", "dt", "en", 456); + } + + assertThat(secondLsn) + .as("WAL LSN must be strictly increasing across restarts") + .isGreaterThan(firstLsn); + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWALMonotonicSegmentTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWALMonotonicSegmentTest.java new file mode 100644 index 00000000000..7c9b563b8ce --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWALMonotonicSegmentTest.java @@ -0,0 +1,137 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ + +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.UUID; +import java.util.regex.Pattern; +import java.util.zip.GZIPOutputStream; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +/** + * Ensures WAL segment numbering remains monotonic across restarts by including gzipped segments when determining the + * next segment sequence. + */ +class ValueStoreWALMonotonicSegmentTest { + + private static final Pattern SEGMENT_GZ = Pattern.compile("wal-(\\d+)\\.v1\\.gz"); + + @TempDir + Path tempDir; + + @Test + void segmentNumberingMonotonicAcrossRestart() throws Exception { + Path walDir = tempDir.resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + Files.createDirectories(walDir); + + ValueStoreWalConfig cfg = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid(UUID.randomUUID().toString()) + .maxSegmentBytes(4096) // small to force rotation and gzip + .build(); + + // 1) Start WAL and generate enough records to produce at least one compressed segment + int minted = 200; + long lastLsn; + try (ValueStoreWAL wal = ValueStoreWAL.open(cfg)) { + lastLsn = mintMany(wal, minted); + wal.awaitDurable(lastLsn); + } + + int beforeMax = maxCompressedSeq(walDir); + assertThat(beforeMax).withFailMessage("Expected at least one gzipped segment after initial rotation") + .isGreaterThanOrEqualTo(1); + + // Ensure there are NO bare segments left before restart, simulating an environment + // where only gzipped segments are present on startup + compressAllBareSegments(walDir); + + // 2) Restart WAL; on open it creates the next bare segment immediately + int expectedNext = maxCompressedSeq(walDir) + 1; + try (ValueStoreWAL wal = ValueStoreWAL.open(cfg)) { + long lsn = wal.logMint(minted + 1, ValueStoreWalValueKind.LITERAL, "restart", "http://example/dt", "", 17); + wal.awaitDurable(lsn); + } + + int openedSeq = currentBareSegmentSeq(walDir); + // The newly opened bare segment must be numbered after the max compressed sequence + // If gz files are ignored when scanning, numbering restarts at 1 + assertThat(openedSeq).isEqualTo(expectedNext); + } + + private static long mintMany(ValueStoreWAL wal, int count) throws IOException { + long lsn = -1; + for (int i = 0; i < count; i++) { + // Minimal payload; IDs and hashes vary to avoid identical frames + lsn = wal.logMint(i + 1, ValueStoreWalValueKind.LITERAL, "lex-" + i, "http://example/dt", "", 31 * i); + } + return lsn; + } + + private static int maxCompressedSeq(Path walDir) throws IOException { + int max = 0; + try (var stream = Files.list(walDir)) { + for (Path path : (Iterable) stream::iterator) { + if (SEGMENT_GZ.matcher(path.getFileName().toString()).matches()) { + int seq = ValueStoreWalTestUtils.readSegmentSequence(path); + if (seq > max) { + max = seq; + } + } + } + } + return max; + } + + private static void compressAllBareSegments(Path walDir) throws IOException { + try (var stream = Files.list(walDir)) { + for (Path p : (Iterable) stream::iterator) { + String name = p.getFileName().toString(); + if (name.startsWith("wal-") && name.endsWith(".v1")) { + Path gz = p.resolveSibling(name + ".gz"); + try (var in = Files.newInputStream(p); + var out = new GZIPOutputStream(Files.newOutputStream(gz))) { + byte[] buf = new byte[1 << 16]; + int r; + while ((r = in.read(buf)) >= 0) { + out.write(buf, 0, r); + } + out.finish(); + } + Files.deleteIfExists(p); + } + } + } + } + + private static int currentBareSegmentSeq(Path walDir) throws IOException { + int seq = 0; + try (var stream = Files.list(walDir)) { + for (Path p : (Iterable) stream::iterator) { + String name = p.getFileName().toString(); + if (name.startsWith("wal-") && name.endsWith(".v1")) { + int current = ValueStoreWalTestUtils.readSegmentSequence(p); + if (current > seq) { + seq = current; + } + } + } + } + return seq; + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWALNoopAndDoubleCloseTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWALNoopAndDoubleCloseTest.java new file mode 100644 index 00000000000..0fee7e4170d --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWALNoopAndDoubleCloseTest.java @@ -0,0 +1,51 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.junit.jupiter.api.Assertions.assertDoesNotThrow; + +import java.nio.file.Path; +import java.util.UUID; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +class ValueStoreWALNoopAndDoubleCloseTest { + + @TempDir + Path tempDir; + + @Test + void awaitDurableNoopForNoLsnAndClosedWal() throws Exception { + ValueStoreWalConfig cfg = ValueStoreWalConfig.builder() + .walDirectory(tempDir) + .storeUuid(UUID.randomUUID().toString()) + .build(); + try (ValueStoreWAL wal = ValueStoreWAL.open(cfg)) { + assertDoesNotThrow(() -> wal.awaitDurable(ValueStoreWAL.NO_LSN)); + } + try (ValueStoreWAL wal = ValueStoreWAL.open(cfg)) { + wal.close(); + assertDoesNotThrow(() -> wal.awaitDurable(123)); + } + } + + @Test + void closeIsIdempotent() throws Exception { + ValueStoreWalConfig cfg = ValueStoreWalConfig.builder() + .walDirectory(tempDir) + .storeUuid(UUID.randomUUID().toString()) + .build(); + ValueStoreWAL wal = ValueStoreWAL.open(cfg); + wal.close(); + assertDoesNotThrow(wal::close); + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWALPurgeWakesProducersTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWALPurgeWakesProducersTest.java new file mode 100644 index 00000000000..934058507af --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWALPurgeWakesProducersTest.java @@ -0,0 +1,327 @@ +/** + ******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.lang.reflect.Field; +import java.lang.reflect.Method; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.UUID; +import java.util.concurrent.BlockingQueue; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +/** + * Verifies that purging the WAL wakes any producers blocked on a full queue. + * + *

    + * Reproduces a deadlock that occurs when {@link java.util.concurrent.ArrayBlockingQueue#clear()} is used during purge: + * it removes elements without signalling {@code notFull}, leaving producers blocked in {@code put()} even though the + * queue is now empty. + */ +class ValueStoreWALPurgeWakesProducersTest { + + @TempDir + Path tempDir; + + @Test + void purgeWakesBlockedProducer() throws Exception { + Path walDir = tempDir.resolve("wal-purge-wakeup"); + Files.createDirectories(walDir); + + ValueStoreWalConfig cfg = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid(UUID.randomUUID().toString()) + .queueCapacity(1) // make saturation easy and deterministic + .build(); + + try (ValueStoreWAL wal = ValueStoreWAL.open(cfg)) { + // Stop the writer thread to avoid it draining the queue during this focused concurrency check + Object logWriter = getField(wal, "logWriter"); + Method shutdown = logWriter.getClass().getDeclaredMethod("shutdown"); + shutdown.setAccessible(true); + shutdown.invoke(logWriter); + + Thread writerThread = (Thread) getField(wal, "writerThread"); + writerThread.join(TimeUnit.SECONDS.toMillis(5)); + assertThat(!writerThread.isAlive()).as("writer thread should be stopped for this test").isTrue(); + + // Swap in a test queue with explicit "clear does not signal notFull" semantics to make the behavior + // deterministic across JDK versions. + TestBlockingQueue testQueue = new TestBlockingQueue(1); + setField(wal, "queue", testQueue); + + // Fill the test queue to capacity so the next mint attempt will block in put() + boolean offered = testQueue.offer(new ValueStoreWalRecord(1L, 1, ValueStoreWalValueKind.LITERAL, + "pre-fill", "dt", "", 0)); + assertThat(offered).isTrue(); + + AtomicBoolean producerFinished = new AtomicBoolean(false); + + Thread producer = new Thread(() -> { + try { + wal.logMint(2, ValueStoreWalValueKind.LITERAL, "after-purge", "dt", "", 0); + producerFinished.set(true); + } catch (Exception e) { + // mark as finished to avoid hanging the test in case of interruption on put() + producerFinished.set(true); + } + }, "blocked-producer"); + + producer.start(); + + // Small delay to ensure the producer is actually blocked on put() + Thread.sleep(50); + assertThat(producer.isAlive()).as("producer should be blocked on a full queue").isTrue(); + + // Perform the purge using the internal method to model the writer's purge path without races + Method performPurge = logWriter.getClass().getDeclaredMethod("performPurgeInternal"); + performPurge.setAccessible(true); + performPurge.invoke(logWriter); + + // Expectation: purge must wake the blocked producer promptly + producer.join(TimeUnit.SECONDS.toMillis(1)); + boolean finishedNaturally = !producer.isAlive(); + try { + assertThat(finishedNaturally) + .as("producer should have completed without external interruption after purge") + .isTrue(); + assertThat(producerFinished.get()) + .as("purge must wake producers blocked in queue.put()") + .isTrue(); + } finally { + if (!finishedNaturally) { + // ensure no stray thread if assertion failed + producer.interrupt(); + } + } + } + } + + private static Object getField(Object target, String name) throws Exception { + Field f = target.getClass().getDeclaredField(name); + f.setAccessible(true); + return f.get(target); + } + + private static void setField(Object target, String name, Object value) throws Exception { + Field f = target.getClass().getDeclaredField(name); + f.setAccessible(true); + f.set(target, value); + } + + /** + * Minimal blocking queue with a fixed capacity whose clear() deliberately does not signal notFull, to reproduce the + * deadlock scenario independent of the JDK's ArrayBlockingQueue implementation. + */ + private static final class TestBlockingQueue implements BlockingQueue { + private final java.util.ArrayDeque deque = new java.util.ArrayDeque<>(); + private final int capacity; + private final java.util.concurrent.locks.ReentrantLock lock = new java.util.concurrent.locks.ReentrantLock(); + private final java.util.concurrent.locks.Condition notEmpty = lock.newCondition(); + private final java.util.concurrent.locks.Condition notFull = lock.newCondition(); + + TestBlockingQueue(int capacity) { + this.capacity = capacity; + } + + @Override + public boolean offer(ValueStoreWalRecord e) { + lock.lock(); + try { + if (deque.size() >= capacity) + return false; + deque.addLast(e); + notEmpty.signal(); + return true; + } finally { + lock.unlock(); + } + } + + @Override + public void put(ValueStoreWalRecord e) throws InterruptedException { + lock.lock(); + try { + while (deque.size() >= capacity) { + notFull.await(); + } + deque.addLast(e); + notEmpty.signal(); + } finally { + lock.unlock(); + } + } + + @Override + public ValueStoreWalRecord poll(long timeout, java.util.concurrent.TimeUnit unit) throws InterruptedException { + long nanos = unit.toNanos(timeout); + lock.lockInterruptibly(); + try { + while (deque.isEmpty()) { + if (nanos <= 0L) + return null; + nanos = notEmpty.awaitNanos(nanos); + } + ValueStoreWalRecord v = deque.removeFirst(); + notFull.signal(); + return v; + } finally { + lock.unlock(); + } + } + + @Override + public void clear() { + lock.lock(); + try { + deque.clear(); + // intentionally do NOT signal notFull here + } finally { + lock.unlock(); + } + } + + @Override + public boolean isEmpty() { + lock.lock(); + try { + return deque.isEmpty(); + } finally { + lock.unlock(); + } + } + + @Override + public int remainingCapacity() { + lock.lock(); + try { + return capacity - deque.size(); + } finally { + lock.unlock(); + } + } + + // --- Methods below are unused in this test and implemented minimally or throw UnsupportedOperationException + // --- + + @Override + public ValueStoreWalRecord take() { + throw new UnsupportedOperationException(); + } + + @Override + public ValueStoreWalRecord poll() { + lock.lock(); + try { + if (deque.isEmpty()) { + return null; + } + ValueStoreWalRecord v = deque.removeFirst(); + notFull.signal(); + return v; + } finally { + lock.unlock(); + } + } + + @Override + public ValueStoreWalRecord remove() { + throw new UnsupportedOperationException(); + } + + @Override + public ValueStoreWalRecord element() { + throw new UnsupportedOperationException(); + } + + @Override + public ValueStoreWalRecord peek() { + return null; + } + + @Override + public boolean add(ValueStoreWalRecord e) { + return offer(e); + } + + @Override + public boolean offer(ValueStoreWalRecord e, long timeout, java.util.concurrent.TimeUnit unit) { + return offer(e); + } + + @Override + public int drainTo(java.util.Collection c) { + throw new UnsupportedOperationException(); + } + + @Override + public int drainTo(java.util.Collection c, int maxElements) { + throw new UnsupportedOperationException(); + } + + @Override + public boolean remove(Object o) { + throw new UnsupportedOperationException(); + } + + @Override + public boolean contains(Object o) { + throw new UnsupportedOperationException(); + } + + @Override + public int size() { + throw new UnsupportedOperationException(); + } + + @Override + public java.util.Iterator iterator() { + throw new UnsupportedOperationException(); + } + + @Override + public Object[] toArray() { + throw new UnsupportedOperationException(); + } + + @Override + public T[] toArray(T[] a) { + throw new UnsupportedOperationException(); + } + + @Override + public boolean containsAll(java.util.Collection c) { + throw new UnsupportedOperationException(); + } + + @Override + public boolean addAll(java.util.Collection c) { + throw new UnsupportedOperationException(); + } + + @Override + public boolean removeAll(java.util.Collection c) { + throw new UnsupportedOperationException(); + } + + @Override + public boolean retainAll(java.util.Collection c) { + throw new UnsupportedOperationException(); + } + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWALReadSegmentSequenceEdgeCasesTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWALReadSegmentSequenceEdgeCasesTest.java new file mode 100644 index 00000000000..0fc149da9e6 --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWALReadSegmentSequenceEdgeCasesTest.java @@ -0,0 +1,90 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.file.Files; +import java.nio.file.Path; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import com.fasterxml.jackson.core.JsonFactory; +import com.fasterxml.jackson.core.JsonGenerator; + +class ValueStoreWALReadSegmentSequenceEdgeCasesTest { + + @TempDir + Path tempDir; + + @Test + void returnsZeroForEmptyOrShortFiles() throws Exception { + Path empty = tempDir.resolve("wal-empty.v1"); + Files.write(empty, new byte[0]); + assertThat(ValueStoreWAL.readSegmentSequence(empty)).isEqualTo(0); + + Path shortHdr = tempDir.resolve("wal-short.v1"); + Files.write(shortHdr, new byte[] { 1, 2 }); + assertThat(ValueStoreWAL.readSegmentSequence(shortHdr)).isEqualTo(0); + } + + @Test + void returnsZeroForNonPositiveLengthAndTruncatedJson() throws Exception { + Path lenZero = tempDir.resolve("wal-lenzero.v1"); + ByteBuffer b = ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putInt(0); + b.flip(); + Files.write(lenZero, b.array()); + assertThat(ValueStoreWAL.readSegmentSequence(lenZero)).isEqualTo(0); + + Path trunc = tempDir.resolve("wal-trunc.v1"); + ByteBuffer hdr = ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putInt(16); + hdr.flip(); + Files.write(trunc, hdr.array()); + assertThat(ValueStoreWAL.readSegmentSequence(trunc)).isEqualTo(0); + } + + @Test + void returnsZeroWhenHeaderHasNoSegmentField() throws Exception { + Path noseg = tempDir.resolve("wal-noseg.v1"); + byte[] json = headerWithoutSegment(); + ByteBuffer out = ByteBuffer.allocate(4 + json.length + 4).order(ByteOrder.LITTLE_ENDIAN); + out.putInt(json.length); + out.put(json); + out.putInt(0); + out.flip(); + byte[] data = new byte[out.remaining()]; + out.get(data); + Files.write(noseg, data); + assertThat(ValueStoreWAL.readSegmentSequence(noseg)).isEqualTo(0); + } + + private static byte[] headerWithoutSegment() throws IOException { + JsonFactory f = new JsonFactory(); + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + try (JsonGenerator g = f.createGenerator(baos)) { + g.writeStartObject(); + g.writeStringField("t", "V"); + g.writeNumberField("ver", 1); + g.writeStringField("store", "s"); + g.writeStringField("engine", "valuestore"); + g.writeNumberField("created", 0); + g.writeNumberField("firstId", 1); + g.writeEndObject(); + } + baos.write('\n'); + return baos.toByteArray(); + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWALRetainPendingForceTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWALRetainPendingForceTest.java new file mode 100644 index 00000000000..787d1769573 --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWALRetainPendingForceTest.java @@ -0,0 +1,83 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import java.lang.reflect.Field; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.UUID; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicLong; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +/** + * Basic sanity for back-to-back awaitDurable calls. This does not attempt to deterministically reproduce the race but + * ensures that in normal use two sequential awaits complete promptly. + */ +class ValueStoreWALRetainPendingForceTest { + + @TempDir + Path tempDir; + + @Test + void backToBackAwaitDoesNotHang() throws Exception { + var walDir = tempDir.resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + Files.createDirectories(walDir); + + ValueStoreWalConfig cfg = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid(UUID.randomUUID().toString()) + .syncPolicy(ValueStoreWalConfig.SyncPolicy.COMMIT) + .build(); + + try (ValueStoreWAL wal = ValueStoreWAL.open(cfg)) { + long lsn1 = wal.logMint(1, ValueStoreWalValueKind.LITERAL, "x", "http://dt", "", 123); + waitUntilLastAppendedAtLeast(wal, lsn1); + + CompletableFuture first = CompletableFuture.runAsync(() -> { + try { + wal.awaitDurable(lsn1); + } catch (Exception e) { + throw new RuntimeException(e); + } + }); + + long lsn2 = wal.logMint(2, ValueStoreWalValueKind.LITERAL, "y", "http://dt", "", 456); + + CompletableFuture second = CompletableFuture.runAsync(() -> { + try { + wal.awaitDurable(lsn2); + } catch (Exception e) { + throw new RuntimeException(e); + } + }); + + CompletableFuture.allOf(first, second).orTimeout(5, TimeUnit.SECONDS).join(); + } + } + + private static void waitUntilLastAppendedAtLeast(ValueStoreWAL wal, long targetLsn) throws Exception { + Field f = ValueStoreWAL.class.getDeclaredField("lastAppendedLsn"); + f.setAccessible(true); + AtomicLong lastAppended = (AtomicLong) f.get(wal); + long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(5); + while (System.nanoTime() < deadline) { + if (lastAppended.get() >= targetLsn) { + return; + } + Thread.sleep(1); + } + throw new AssertionError("writer thread did not append record in time"); + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalBootstrapResumeTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalBootstrapResumeTest.java new file mode 100644 index 00000000000..f1e89f998b7 --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalBootstrapResumeTest.java @@ -0,0 +1,111 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import java.nio.file.Files; +import java.nio.file.Path; +import java.time.Duration; +import java.util.UUID; +import java.util.concurrent.TimeUnit; + +import org.eclipse.rdf4j.model.IRI; +import org.eclipse.rdf4j.model.impl.SimpleValueFactory; +import org.eclipse.rdf4j.sail.nativerdf.ValueStore; +import org.eclipse.rdf4j.sail.nativerdf.datastore.DataStore; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.Timeout; +import org.junit.jupiter.api.io.TempDir; + +/** + * Verifies that ValueStore resumes WAL bootstrap after a partial prior run (segments exist but no completion marker). + */ +public class ValueStoreWalBootstrapResumeTest { + + @TempDir + Path tmp; + + @Test + @Timeout(value = 5, unit = TimeUnit.MINUTES) + void resumesBootstrapAfterPartialRun() throws Exception { + // 1) Create a ValueStore with some existing values, without WAL + Path data = tmp.resolve("data"); + Files.createDirectories(data); + try (ValueStore vs = new ValueStore(data.toFile())) { + // Create enough values to ensure bootstrap takes noticeable time + for (int i = 0; i < 5000; i++) { + IRI v = SimpleValueFactory.getInstance().createIRI("urn:test:" + i); + vs.storeValue(v); + } + vs.sync(); + } + + // 2) Open with WAL enabled (async bootstrap), let it start and create at least one segment + Path walDir = tmp.resolve("wal"); + Files.createDirectories(walDir); + ValueStoreWalConfig cfg = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid("test-" + UUID.randomUUID()) + .syncPolicy(ValueStoreWalConfig.SyncPolicy.ALWAYS) + .syncBootstrapOnOpen(false) + .build(); + + ValueStoreWAL wal = ValueStoreWAL.open(cfg); + ValueStore vs2 = new ValueStore(data.toFile(), false, + ValueStore.VALUE_CACHE_SIZE, ValueStore.VALUE_ID_CACHE_SIZE, + ValueStore.NAMESPACE_CACHE_SIZE, ValueStore.NAMESPACE_ID_CACHE_SIZE, + wal); + + // Simulate sudden crash by closing WAL directly almost immediately; bootstrap + // thread will observe isClosed and stop early (partial run or none) + Thread.sleep(5); + wal.close(); + vs2.close(); + + // 3) Reopen with WAL; after partial run, resume bootstrap and bring WAL dictionary + // to cover all existing ValueStore IDs. + int expectedMaxId; + try (DataStore ds = new DataStore(data.toFile(), "values")) { + expectedMaxId = ds.getMaxID(); + } + + try (ValueStoreWAL wal2 = ValueStoreWAL.open(cfg); + ValueStore vs3 = new ValueStore(data.toFile(), false, + ValueStore.VALUE_CACHE_SIZE, ValueStore.VALUE_ID_CACHE_SIZE, + ValueStore.NAMESPACE_CACHE_SIZE, ValueStore.NAMESPACE_ID_CACHE_SIZE, + wal2)) { + + waitUntil(() -> { + try (ValueStoreWalReader reader = ValueStoreWalReader.open(cfg)) { + ValueStoreWalRecovery.ReplayReport report = new ValueStoreWalRecovery().replayWithReport(reader); + return report.dictionary().size() == expectedMaxId; + } + }, Duration.ofSeconds(120)); + } + } + + private static void waitUntil(Condition cond, Duration timeout) throws Exception { + long deadline = System.nanoTime() + timeout.toNanos(); + while (System.nanoTime() < deadline) { + if (cond.ok()) + return; + Thread.sleep(20); + } + // one last check before failing + if (!cond.ok()) { + throw new AssertionError("Condition not met within timeout: " + timeout); + } + } + + @FunctionalInterface + private interface Condition { + boolean ok() throws Exception; + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalClearPurgeTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalClearPurgeTest.java new file mode 100644 index 00000000000..a5210bcef22 --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalClearPurgeTest.java @@ -0,0 +1,84 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.File; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.OptionalLong; +import java.util.UUID; + +import org.eclipse.rdf4j.model.IRI; +import org.eclipse.rdf4j.model.ValueFactory; +import org.eclipse.rdf4j.model.impl.SimpleValueFactory; +import org.eclipse.rdf4j.sail.nativerdf.ValueStore; +import org.eclipse.rdf4j.sail.nativerdf.model.NativeValue; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +class ValueStoreWalClearPurgeTest { + + private static final ValueFactory VF = SimpleValueFactory.getInstance(); + + @TempDir + Path tempDir; + + @Test + void clearMustPurgeWalToPreventResurrection() throws Exception { + Path walDir = tempDir.resolve("wal-clear"); + Files.createDirectories(walDir); + + ValueStoreWalConfig config = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid(UUID.randomUUID().toString()) + .recoverValueStoreOnOpen(true) + .build(); + + IRI iri = VF.createIRI("http://example.com/resurrect-me"); + + File valuesDir = tempDir.resolve("values-clear").toFile(); + Files.createDirectories(valuesDir.toPath()); + + // Write a value and ensure it is durably logged in the WAL + try (ValueStoreWAL wal = ValueStoreWAL.open(config); + ValueStore store = new ValueStore(valuesDir, false, + ValueStore.VALUE_CACHE_SIZE, + ValueStore.VALUE_ID_CACHE_SIZE, + ValueStore.NAMESPACE_CACHE_SIZE, + ValueStore.NAMESPACE_ID_CACHE_SIZE, + wal)) { + store.storeValue(iri); + OptionalLong lsn = store.drainPendingWalHighWaterMark(); + assertThat(lsn).isPresent(); + wal.awaitDurable(lsn.getAsLong()); + + // Now clear the value store + store.clear(); + } + + // Simulate restart with recovery enabled: if WAL was not purged on clear(), + // recovery would resurrect the value into an otherwise empty store. + try (ValueStoreWAL wal2 = ValueStoreWAL.open(config); + ValueStore store2 = new ValueStore(valuesDir, false, + ValueStore.VALUE_CACHE_SIZE, + ValueStore.VALUE_ID_CACHE_SIZE, + ValueStore.NAMESPACE_CACHE_SIZE, + ValueStore.NAMESPACE_ID_CACHE_SIZE, + wal2)) { + int id = store2.getID(iri); + assertThat(id) + .as("After clear() the WAL must not resurrect deleted values upon recovery") + .isEqualTo(NativeValue.UNKNOWN_ID); + } + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalCombinatoricsTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalCombinatoricsTest.java new file mode 100644 index 00000000000..757f83f56a2 --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalCombinatoricsTest.java @@ -0,0 +1,232 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ + +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.time.Duration; +import java.util.ArrayList; +import java.util.EnumSet; +import java.util.List; +import java.util.UUID; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.stream.Collectors; +import java.util.stream.Stream; + +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.TestInstance; +import org.junit.jupiter.api.TestInstance.Lifecycle; +import org.junit.jupiter.api.io.TempDir; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.Arguments; +import org.junit.jupiter.params.provider.MethodSource; + +/** + * Parameterized combinatorics tests that exercise the ValueStore WAL writer under a variety of sync, durability and + * purge permutations. The goal is to ensure that no matter which combination is chosen the WAL produces a consistent, + * monotonically ordered set of records without leaking stale segments. + */ +@TestInstance(Lifecycle.PER_CLASS) +class ValueStoreWalCombinatoricsTest { + + private static final Duration SYNC_INTERVAL = Duration.ofMillis(2); + private static final Duration IDLE_POLL_INTERVAL = Duration.ofMillis(1); + private static final long MAX_SEGMENT_BYTES = 2048; + private static final int BATCH_BUFFER_BYTES = 1 << 15; + private static final int QUEUE_CAPACITY = 16; + private static final int SEED_RECORDS = 24; + + private enum ForceMode { + NONE, + FINAL, + EACH + } + + private enum PurgeMode { + NEVER, + MID_STREAM + } + + private enum InitialState { + EMPTY, + SEEDED + } + + @TempDir + Path tempDir; + + private final AtomicInteger idCounter = new AtomicInteger(); + private String storeUuid; + + @BeforeEach + void setUp() { + storeUuid = UUID.randomUUID().toString(); + } + + @AfterEach + void tearDown() { + idCounter.set(0); + } + + @ParameterizedTest(name = "{index}: policy={0}, force={1}, purge={2}, seed={3}") + @MethodSource("walCombinationCases") + void walHandlesCombinations(ValueStoreWalConfig.SyncPolicy syncPolicy, ForceMode forceMode, PurgeMode purgeMode, + InitialState initialState) throws Exception { + + Path walDir = createWalDirectory(syncPolicy, forceMode, purgeMode, initialState); + + List expectedLexicals = new ArrayList<>(); + if (initialState == InitialState.SEEDED) { + expectedLexicals.addAll(seedInitialSegments(walDir)); + } + + ValueStoreWalConfig config = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid(storeUuid) + .maxSegmentBytes(MAX_SEGMENT_BYTES) + .queueCapacity(QUEUE_CAPACITY) + .batchBufferBytes(BATCH_BUFFER_BYTES) + .syncPolicy(syncPolicy) + .syncInterval(SYNC_INTERVAL) + .idlePollInterval(IDLE_POLL_INTERVAL) + .build(); + + try (ValueStoreWAL wal = ValueStoreWAL.open(config)) { + BatchResult firstBatch = mintBatch(wal, "first", 6, forceMode); + expectedLexicals.addAll(firstBatch.lexicals()); + + if (purgeMode == PurgeMode.MID_STREAM) { + wal.purgeAllSegments(); + expectedLexicals.clear(); + } + + BatchResult secondBatch = mintBatch(wal, "second", 5, forceMode); + expectedLexicals.addAll(secondBatch.lexicals()); + } + + ValueStoreWalReader.ScanResult result = ValueStoreWalReader.open(config).scan(); + List actualLexicals = result.records() + .stream() + .map(ValueStoreWalRecord::lexical) + .collect(Collectors.toList()); + + assertThat(actualLexicals).containsExactlyElementsOf(expectedLexicals); + if (purgeMode == PurgeMode.NEVER) { + assertThat(result.complete()) + .as("WAL scan should be complete when no purge occurs") + .isTrue(); + } + + List lsns = result.records() + .stream() + .map(ValueStoreWalRecord::lsn) + .collect(Collectors.toList()); + for (int i = 1; i < lsns.size(); i++) { + assertThat(lsns.get(i)).isGreaterThan(lsns.get(i - 1)); + } + + if (expectedLexicals.isEmpty()) { + assertThat(lsns).isEmpty(); + assertThat(result.lastValidLsn()).isEqualTo(ValueStoreWAL.NO_LSN); + } else { + assertThat(lsns).isNotEmpty(); + assertThat(result.lastValidLsn()).isGreaterThanOrEqualTo(lsns.get(lsns.size() - 1)); + } + } + + private Stream walCombinationCases() { + List arguments = new ArrayList<>(); + for (ValueStoreWalConfig.SyncPolicy policy : ValueStoreWalConfig.SyncPolicy.values()) { + for (ForceMode forceMode : ForceMode.values()) { + for (PurgeMode purgeMode : PurgeMode.values()) { + for (InitialState seed : EnumSet.allOf(InitialState.class)) { + arguments.add(Arguments.of(policy, forceMode, purgeMode, seed)); + } + } + } + } + return arguments.stream(); + } + + private Path createWalDirectory(ValueStoreWalConfig.SyncPolicy syncPolicy, ForceMode forceMode, + PurgeMode purgeMode, InitialState seed) throws IOException { + String dirName = (syncPolicy.name() + "-" + forceMode.name() + "-" + purgeMode.name() + "-" + seed.name()) + .toLowerCase(); + Path dir = tempDir.resolve(dirName); + Files.createDirectories(dir); + return dir; + } + + private List seedInitialSegments(Path walDir) throws Exception { + ValueStoreWalConfig seedConfig = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid(storeUuid) + .maxSegmentBytes(MAX_SEGMENT_BYTES) + .queueCapacity(QUEUE_CAPACITY) + .batchBufferBytes(BATCH_BUFFER_BYTES) + .syncPolicy(ValueStoreWalConfig.SyncPolicy.ALWAYS) + .syncInterval(SYNC_INTERVAL) + .idlePollInterval(IDLE_POLL_INTERVAL) + .build(); + + try (ValueStoreWAL wal = ValueStoreWAL.open(seedConfig)) { + return new ArrayList<>(mintBatch(wal, "seed", SEED_RECORDS, ForceMode.FINAL).lexicals()); + } + } + + private BatchResult mintBatch(ValueStoreWAL wal, String prefix, int count, ForceMode forceMode) + throws IOException, InterruptedException { + List lexicals = new ArrayList<>(count); + long lastLsn = ValueStoreWAL.NO_LSN; + for (int i = 0; i < count; i++) { + int id = idCounter.incrementAndGet(); + String lexical = lexicalToken(prefix, id); + long lsn = wal.logMint(id, ValueStoreWalValueKind.LITERAL, lexical, "http://example/dt", "", + lexical.hashCode()); + lexicals.add(lexical); + if (forceMode == ForceMode.EACH) { + wal.awaitDurable(lsn); + } + lastLsn = lsn; + } + if (forceMode == ForceMode.FINAL && lastLsn > ValueStoreWAL.NO_LSN) { + wal.awaitDurable(lastLsn); + } + return new BatchResult(lexicals, lastLsn); + } + + private static String lexicalToken(String prefix, int id) { + return prefix + "-" + id + "-payload-0123456789abcdefghijklmnopqrstuvwxyz"; + } + + private static final class BatchResult { + private final List lexicals; + private final long lastLsn; + + private BatchResult(List lexicals, long lastLsn) { + this.lexicals = List.copyOf(lexicals); + this.lastLsn = lastLsn; + } + + private List lexicals() { + return lexicals; + } + + private long lastLsn() { + return lastLsn; + } + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalCompressedNoSummaryTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalCompressedNoSummaryTest.java new file mode 100644 index 00000000000..5265d1a0bcf --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalCompressedNoSummaryTest.java @@ -0,0 +1,109 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.zip.CRC32C; +import java.util.zip.GZIPOutputStream; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import com.fasterxml.jackson.core.JsonFactory; +import com.fasterxml.jackson.core.JsonGenerator; + +/** + * Validates that a compressed segment lacking a summary frame results in incomplete scan. + */ +class ValueStoreWalCompressedNoSummaryTest { + + @TempDir + Path tempDir; + + @Test + void compressedSegmentWithoutSummaryMarksIncomplete() throws Exception { + Path walDir = tempDir.resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + Files.createDirectories(walDir); + Path gz = walDir.resolve("wal-1.v1.gz"); + + try (GZIPOutputStream out = new GZIPOutputStream(Files.newOutputStream(gz))) { + // Header frame + frame(out, headerJson(1, 1)); + // One minted frame + frame(out, mintedJson(1L, 1)); + // No summary frame + out.finish(); + } + + ValueStoreWalConfig cfg = ValueStoreWalConfig.builder().walDirectory(walDir).storeUuid("s").build(); + try (ValueStoreWalReader reader = ValueStoreWalReader.open(cfg)) { + ValueStoreWalReader.ScanResult res = reader.scan(); + assertThat(res.complete()).isFalse(); + assertThat(res.records()).hasSize(1); + } + } + + private static void frame(GZIPOutputStream out, byte[] json) throws IOException { + ByteBuffer lb = ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putInt(json.length); + lb.flip(); + out.write(lb.array(), 0, 4); + out.write(json); + CRC32C c = new CRC32C(); + c.update(json, 0, json.length); + ByteBuffer cb = ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putInt((int) c.getValue()); + cb.flip(); + out.write(cb.array(), 0, 4); + } + + private static byte[] headerJson(int segment, int firstId) throws IOException { + JsonFactory f = new JsonFactory(); + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + try (JsonGenerator g = f.createGenerator(baos)) { + g.writeStartObject(); + g.writeStringField("t", "V"); + g.writeNumberField("ver", 1); + g.writeStringField("store", "s"); + g.writeStringField("engine", "valuestore"); + g.writeNumberField("created", 0); + g.writeNumberField("segment", segment); + g.writeNumberField("firstId", firstId); + g.writeEndObject(); + } + baos.write('\n'); + return baos.toByteArray(); + } + + private static byte[] mintedJson(long lsn, int id) throws IOException { + JsonFactory f = new JsonFactory(); + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + try (JsonGenerator g = f.createGenerator(baos)) { + g.writeStartObject(); + g.writeStringField("t", "M"); + g.writeNumberField("lsn", lsn); + g.writeNumberField("id", id); + g.writeStringField("vk", "I"); + g.writeStringField("lex", "http://ex/id" + id); + g.writeStringField("dt", ""); + g.writeStringField("lang", ""); + g.writeNumberField("hash", 0); + g.writeEndObject(); + } + baos.write('\n'); + return baos.toByteArray(); + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalCompressedSegmentRestoreTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalCompressedSegmentRestoreTest.java new file mode 100644 index 00000000000..b96894371ea --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalCompressedSegmentRestoreTest.java @@ -0,0 +1,299 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ + +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.IOException; +import java.io.InputStream; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.Comparator; +import java.util.List; +import java.util.UUID; +import java.util.regex.Matcher; +import java.util.regex.Pattern; +import java.util.zip.CRC32C; +import java.util.zip.GZIPInputStream; + +import org.eclipse.rdf4j.model.ValueFactory; +import org.eclipse.rdf4j.model.impl.SimpleValueFactory; +import org.eclipse.rdf4j.sail.nativerdf.ValueStore; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import com.fasterxml.jackson.core.JsonFactory; +import com.fasterxml.jackson.core.JsonParser; +import com.fasterxml.jackson.core.JsonToken; + +/** + * Restores a value record from a compressed ValueStore WAL segment by performing a binary search on segment first LSNs. + */ +class ValueStoreWalCompressedSegmentRestoreTest { + + private static final ValueFactory VF = SimpleValueFactory.getInstance(); + private static final Pattern SEGMENT_GZ = Pattern.compile("wal-(\\d+)\\.v1\\.gz"); + + @TempDir + Path tempDir; + + @Test + void restoreFromCompressedSegmentUsingBinarySearch() throws Exception { + // Force multiple segments by limiting segment size + Path walDir = tempDir.resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + Files.createDirectories(walDir); + ValueStoreWalConfig config = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid(UUID.randomUUID().toString()) + .maxSegmentBytes(4096) // small to ensure rotation + gzip + .build(); + + // Write enough values to rotate segments + String targetLex = null; + long targetLsn = -1; + try (ValueStoreWAL wal = ValueStoreWAL.open(config)) { + Path valuesDir = tempDir.resolve("values"); + Files.createDirectories(valuesDir); + try (ValueStore store = new ValueStore(valuesDir.toFile(), false, ValueStore.VALUE_CACHE_SIZE, + ValueStore.VALUE_ID_CACHE_SIZE, ValueStore.NAMESPACE_CACHE_SIZE, + ValueStore.NAMESPACE_ID_CACHE_SIZE, wal)) { + // Mint many literal values to span several segments + for (int i = 0; i < 1000; i++) { + String lex = "val-" + i; + store.storeValue(VF.createLiteral(lex)); + var lsn = store.drainPendingWalHighWaterMark(); + if (i == 123) { // pick an early target to likely land in a compressed segment + targetLex = lex; + targetLsn = lsn.orElse(-1); + } + } + wal.awaitDurable(targetLsn); + } + } + + // Ensure we have compressed segments + List compressed = listCompressedSegments(walDir); + assertThat(compressed).isNotEmpty(); + + // Compute first LSN per compressed segment (first 'M' after header) + List firstLsns = new ArrayList<>(compressed.size()); + for (Path gz : compressed) { + long first = firstMintLsn(gz); + firstLsns.add(first); + } + + // If our chosen target ended up after compressed segments, pick a target inside compressed range + long maxFirst = firstLsns.get(firstLsns.size() - 1); + if (targetLsn <= 0 || targetLsn < firstLsns.get(0) || targetLsn >= maxFirst) { + // fallback: derive a target from within first compressed segment by scanning a few frames + Target t = pickTargetFromCompressed(compressed.get(0)); + targetLex = t.lex; + targetLsn = t.lsn; + } + + // Binary search compressed segments by their first LSN + int segIdx = lowerBound(firstLsns, targetLsn); + if (segIdx == firstLsns.size() || firstLsns.get(segIdx) > targetLsn) { + segIdx = Math.max(0, segIdx - 1); + } + Path candidate = compressed.get(segIdx); + + // Scan the candidate compressed segment to find our target and restore its lexical + ValueStoreWalRecord rec = scanSegmentForLsn(candidate, targetLsn); + assertThat(rec).withFailMessage("target LSN not found in compressed segment").isNotNull(); + assertThat(rec.lexical()).isEqualTo(targetLex); + } + + private static int lowerBound(List firstLsns, long lsn) { + int lo = 0, hi = firstLsns.size(); + while (lo < hi) { + int mid = (lo + hi) >>> 1; + if (firstLsns.get(mid) <= lsn) { + lo = mid + 1; + } else { + hi = mid; + } + } + return lo; + } + + private static List listCompressedSegments(Path walDir) throws IOException { + class Item { + final Path path; + final long firstId; + + Item(Path path, long firstId) { + this.path = path; + this.firstId = firstId; + } + } + List items = new ArrayList<>(); + try (var stream = Files.list(walDir)) { + stream.forEach(p -> { + Matcher m = SEGMENT_GZ.matcher(p.getFileName().toString()); + if (m.matches()) { + long firstId = Long.parseLong(m.group(1)); + items.add(new Item(p, firstId)); + } + }); + } + items.sort(Comparator.comparingLong(it -> it.firstId)); + List segments = new ArrayList<>(items.size()); + for (Item item : items) { + segments.add(item.path); + } + return segments; + } + + private static long firstMintLsn(Path gz) throws IOException { + try (GZIPInputStream in = new GZIPInputStream(Files.newInputStream(gz))) { + // header frame + int headerLen = readIntLE(in); + if (headerLen <= 0) { + return -1; + } + byte[] header = in.readNBytes(headerLen); + if (header.length < headerLen) + return -1; + readIntLE(in); // header CRC + // first mint frame + int len = readIntLE(in); + byte[] json = in.readNBytes(len); + readIntLE(in); // crc + Parsed p = parseJson(json); + return p.lsn; + } + } + + private static ValueStoreWalRecord scanSegmentForLsn(Path gz, long targetLsn) throws IOException { + try (GZIPInputStream in = new GZIPInputStream(Files.newInputStream(gz))) { + // skip header + int headerLen = readIntLE(in); + if (headerLen <= 0) + return null; + byte[] header = in.readNBytes(headerLen); + if (header.length < headerLen) + return null; + readIntLE(in); + // scan records + while (true) { + int length = readIntLE(in); + if (length <= 0) + return null; + byte[] jsonBytes = in.readNBytes(length); + if (jsonBytes.length < length) + return null; + int expected = readIntLE(in); + CRC32C crc = new CRC32C(); + crc.update(jsonBytes, 0, jsonBytes.length); + if ((int) crc.getValue() != expected) + return null; + Parsed p = parseJson(jsonBytes); + if (p.type == 'M' && p.lsn == targetLsn) { + return new ValueStoreWalRecord(p.lsn, p.id, p.kind, p.lex, p.dt, p.lang, p.hash); + } + } + } + } + + private static int readIntLE(InputStream in) throws IOException { + byte[] b = in.readNBytes(4); + if (b.length < 4) + return -1; + return ((b[0] & 0xFF)) | ((b[1] & 0xFF) << 8) | ((b[2] & 0xFF) << 16) | ((b[3] & 0xFF) << 24); + } + + private static final JsonFactory JSON_FACTORY = new JsonFactory(); + + private static Parsed parseJson(byte[] jsonBytes) throws IOException { + Parsed parsed = new Parsed(); + try (JsonParser jp = JSON_FACTORY.createParser(jsonBytes)) { + if (jp.nextToken() != JsonToken.START_OBJECT) { + return parsed; + } + while (jp.nextToken() != JsonToken.END_OBJECT) { + String field = jp.getCurrentName(); + jp.nextToken(); + if ("t".equals(field)) { + String t = jp.getValueAsString(""); + parsed.type = t.isEmpty() ? '?' : t.charAt(0); + } else if ("lsn".equals(field)) { + parsed.lsn = jp.getValueAsLong(ValueStoreWAL.NO_LSN); + } else if ("id".equals(field)) { + parsed.id = jp.getValueAsInt(0); + } else if ("vk".equals(field)) { + String code = jp.getValueAsString(""); + parsed.kind = ValueStoreWalValueKind.fromCode(code); + } else if ("lex".equals(field)) { + parsed.lex = jp.getValueAsString(""); + } else if ("dt".equals(field)) { + parsed.dt = jp.getValueAsString(""); + } else if ("lang".equals(field)) { + parsed.lang = jp.getValueAsString(""); + } else if ("hash".equals(field)) { + parsed.hash = jp.getValueAsInt(0); + } else { + jp.skipChildren(); + } + } + } + return parsed; + } + + private static final class Parsed { + char type = '?'; + long lsn = ValueStoreWAL.NO_LSN; + int id = 0; + ValueStoreWalValueKind kind = ValueStoreWalValueKind.NAMESPACE; + String lex = ""; + String dt = ""; + String lang = ""; + int hash = 0; + } + + private static final class Target { + final long lsn; + final String lex; + + Target(long lsn, String lex) { + this.lsn = lsn; + this.lex = lex; + } + } + + private static Target pickTargetFromCompressed(Path gz) throws IOException { + try (GZIPInputStream in = new GZIPInputStream(Files.newInputStream(gz))) { + // skip header + int headerLen = readIntLE(in); + if (headerLen <= 0) + return new Target(-1, ""); + byte[] header = in.readNBytes(headerLen); + if (header.length < headerLen) + return new Target(-1, ""); + readIntLE(in); + // read a couple of mint records and pick the second one + // first mint + int len1 = readIntLE(in); + byte[] j1 = in.readNBytes(len1); + readIntLE(in); + Parsed p1 = parseJson(j1); + // second mint (likely a user value) + int len2 = readIntLE(in); + byte[] j2 = in.readNBytes(len2); + readIntLE(in); + Parsed p2 = parseJson(j2); + Parsed chosen = p2.type == 'M' ? p2 : p1; + return new Target(chosen.lsn, chosen.lex); + } + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalCompressedSummaryCrcValidationTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalCompressedSummaryCrcValidationTest.java new file mode 100644 index 00000000000..71010212ba2 --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalCompressedSummaryCrcValidationTest.java @@ -0,0 +1,196 @@ +/** + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + */ +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.UUID; +import java.util.regex.Pattern; +import java.util.zip.CRC32C; +import java.util.zip.GZIPInputStream; +import java.util.zip.GZIPOutputStream; + +import org.eclipse.rdf4j.model.impl.SimpleValueFactory; +import org.eclipse.rdf4j.sail.nativerdf.ValueStore; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import com.fasterxml.jackson.core.JsonFactory; +import com.fasterxml.jackson.core.JsonGenerator; +import com.fasterxml.jackson.core.JsonParser; +import com.fasterxml.jackson.core.JsonToken; + +/** + * Validates that ValueStoreWalReader verifies the CRC32 summary embedded in compressed segments and marks the scan as + * incomplete when the summary does not match the decompressed content. + */ +class ValueStoreWalCompressedSummaryCrcValidationTest { + + private static final Pattern SEGMENT_GZ = Pattern.compile("wal-(\\d+)\\.v1\\.gz"); + + @TempDir + Path tempDir; + + @Test + void mismatchSummaryCrcMarksScanIncomplete() throws Exception { + // Arrange: create a WAL with at least one compressed segment + Path walDir = tempDir.resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + Files.createDirectories(walDir); + ValueStoreWalConfig config = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid(UUID.randomUUID().toString()) + .maxSegmentBytes(4096) // small to ensure rotation + gzip + .build(); + + // Write enough values to rotate segments and compress the first + try (ValueStoreWAL wal = ValueStoreWAL.open(config)) { + Path valuesDir = tempDir.resolve("values"); + Files.createDirectories(valuesDir); + try (ValueStore store = new ValueStore(valuesDir.toFile(), false, ValueStore.VALUE_CACHE_SIZE, + ValueStore.VALUE_ID_CACHE_SIZE, ValueStore.NAMESPACE_CACHE_SIZE, + ValueStore.NAMESPACE_ID_CACHE_SIZE, wal)) { + var vf = SimpleValueFactory.getInstance(); + for (int i = 0; i < 1000; i++) { + store.storeValue(vf.createLiteral("val-" + i)); + } + } + } + + // Pick one compressed segment + Path compressed = locateFirstCompressed(walDir); + assertThat(compressed).as("compressed WAL segment").isNotNull(); + + // Corrupt the summary CRC inside the compressed segment while keeping per-frame CRCs valid + corruptSummaryCrc32(compressed); + + // Act: scan with reader + try (ValueStoreWalReader reader = ValueStoreWalReader.open(config)) { + ValueStoreWalReader.ScanResult scan = reader.scan(); + // Assert: the scan is marked incomplete due to summary CRC mismatch + assertThat(scan.complete()).as("scan completeness should be false if summary CRC mismatches").isFalse(); + } + } + + private static Path locateFirstCompressed(Path walDir) throws IOException { + try (var stream = Files.list(walDir)) { + return stream.filter(p -> SEGMENT_GZ.matcher(p.getFileName().toString()).matches()) + .findFirst() + .orElse(null); + } + } + + private static void corruptSummaryCrc32(Path gz) throws IOException { + // Decompress entire segment + byte[] decompressed; + try (GZIPInputStream gin = new GZIPInputStream(Files.newInputStream(gz))) { + decompressed = gin.readAllBytes(); + } + + // Walk frames to find the summary frame and its start offset + int pos = 0; + int summaryOffset = -1; + int lastId = 0; + while (pos + 12 <= decompressed.length) { // need at least len + crc around data + int length = getIntLE(decompressed, pos); + pos += 4; + if (pos + length + 4 > decompressed.length) { + break; // truncated safeguard + } + byte[] json = new byte[length]; + System.arraycopy(decompressed, pos, json, 0, length); + pos += length; + // skip frame CRC32C + pos += 4; + + // Parse JSON and detect summary frame + try (JsonParser jp = new JsonFactory().createParser(json)) { + if (jp.nextToken() != JsonToken.START_OBJECT) { + continue; + } + String type = null; + Integer lid = null; + while (jp.nextToken() != JsonToken.END_OBJECT) { + String field = jp.getCurrentName(); + jp.nextToken(); + if ("t".equals(field)) { + type = jp.getValueAsString(""); + } else if ("lastId".equals(field)) { + lid = jp.getValueAsInt(0); + } else { + jp.skipChildren(); + } + } + if ("S".equals(type)) { + summaryOffset = pos - (length + 4 /* len */ + 4 /* crc */); + lastId = lid == null ? 0 : lid.intValue(); + break; + } + } + } + + if (summaryOffset < 0) { + throw new IOException("No summary frame found in compressed WAL segment: " + gz); + } + + // Original content without the summary frame + byte[] originalWithoutSummary = new byte[summaryOffset]; + System.arraycopy(decompressed, 0, originalWithoutSummary, 0, summaryOffset); + + // Build replacement summary frame with deliberately wrong crc32 value + byte[] newSummary = buildSummaryFrameWithCrc(lastId, 0L); // mismatch on purpose + + // Rebuild gz with intact content and corrupted summary + try (GZIPOutputStream gout = new GZIPOutputStream(Files.newOutputStream(gz))) { + gout.write(originalWithoutSummary); + gout.write(newSummary); + gout.finish(); + } + } + + private static byte[] buildSummaryFrameWithCrc(int lastMintedId, long wrongCrc32) throws IOException { + JsonFactory factory = new JsonFactory(); + ByteArrayOutputStream baos = new ByteArrayOutputStream(128); + try (JsonGenerator gen = factory.createGenerator(baos)) { + gen.writeStartObject(); + gen.writeStringField("t", "S"); + gen.writeNumberField("lastId", lastMintedId); + gen.writeNumberField("crc32", wrongCrc32 & 0xFFFFFFFFL); + gen.writeEndObject(); + } + baos.write('\n'); + byte[] json = baos.toByteArray(); + + // Frame = lenLE + json + crc32cLE(json) + ByteBuffer lenBuf = ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putInt(json.length); + CRC32C crc32c = new CRC32C(); + crc32c.update(json, 0, json.length); + int crc = (int) crc32c.getValue(); + ByteBuffer crcBuf = ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putInt(crc); + lenBuf.flip(); + crcBuf.flip(); + byte[] framed = new byte[4 + json.length + 4]; + lenBuf.get(framed, 0, 4); + System.arraycopy(json, 0, framed, 4, json.length); + crcBuf.get(framed, 4 + json.length, 4); + return framed; + } + + private static int getIntLE(byte[] arr, int off) { + return (arr[off] & 0xFF) | ((arr[off + 1] & 0xFF) << 8) | ((arr[off + 2] & 0xFF) << 16) + | ((arr[off + 3] & 0xFF) << 24); + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalConfigValidationTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalConfigValidationTest.java new file mode 100644 index 00000000000..811c73e09dd --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalConfigValidationTest.java @@ -0,0 +1,66 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +import java.nio.file.Path; +import java.util.UUID; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +class ValueStoreWalConfigValidationTest { + + @TempDir + Path tempDir; + + @Test + void requiresWalDirectory() { + ValueStoreWalConfig.Builder b = ValueStoreWalConfig.builder().storeUuid(UUID.randomUUID().toString()); + assertThatThrownBy(b::build).isInstanceOf(IllegalStateException.class) + .hasMessageContaining("walDirectory"); + } + + @Test + void requiresStoreUuid() { + ValueStoreWalConfig.Builder b = ValueStoreWalConfig.builder().walDirectory(tempDir); + assertThatThrownBy(b::build).isInstanceOf(IllegalStateException.class) + .hasMessageContaining("storeUuid"); + } + + @Test + void validatesPositiveSizes() { + // maxSegmentBytes must be > 0 + ValueStoreWalConfig.Builder base1 = ValueStoreWalConfig.builder() + .walDirectory(tempDir) + .storeUuid(UUID.randomUUID().toString()); + assertThatThrownBy(() -> base1.maxSegmentBytes(0).build()) + .isInstanceOf(IllegalStateException.class) + .hasMessageContaining("maxSegmentBytes"); + + // queueCapacity must be > 0 + ValueStoreWalConfig.Builder base2 = ValueStoreWalConfig.builder() + .walDirectory(tempDir) + .storeUuid(UUID.randomUUID().toString()); + assertThatThrownBy(() -> base2.queueCapacity(0).build()) + .isInstanceOf(IllegalStateException.class) + .hasMessageContaining("queueCapacity"); + + // batchBufferBytes must be > 4KB + ValueStoreWalConfig.Builder base3 = ValueStoreWalConfig.builder() + .walDirectory(tempDir) + .storeUuid(UUID.randomUUID().toString()); + assertThatThrownBy(() -> base3.batchBufferBytes(4096).build()) + .isInstanceOf(IllegalStateException.class) + .hasMessageContaining("batchBufferBytes"); + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalCorruptRecoveryTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalCorruptRecoveryTest.java new file mode 100644 index 00000000000..124a82c11ed --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalCorruptRecoveryTest.java @@ -0,0 +1,433 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ + +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.File; +import java.io.RandomAccessFile; +import java.math.BigDecimal; +import java.math.BigInteger; +import java.nio.file.Files; +import java.nio.file.Path; +import java.time.Duration; +import java.time.LocalDate; +import java.time.LocalDateTime; +import java.time.LocalTime; +import java.time.OffsetDateTime; +import java.time.Period; +import java.time.ZoneOffset; +import java.util.Date; +import java.util.OptionalLong; +import java.util.UUID; +import java.util.stream.Stream; + +import javax.xml.datatype.DatatypeFactory; +import javax.xml.datatype.XMLGregorianCalendar; + +import org.eclipse.rdf4j.model.Literal; +import org.eclipse.rdf4j.model.ValueFactory; +import org.eclipse.rdf4j.model.base.CoreDatatype; +import org.eclipse.rdf4j.model.impl.SimpleValueFactory; +import org.eclipse.rdf4j.sail.nativerdf.NativeStore; +import org.eclipse.rdf4j.sail.nativerdf.ValueStore; +import org.eclipse.rdf4j.sail.nativerdf.datastore.IDFile; +import org.eclipse.rdf4j.sail.nativerdf.model.NativeLiteral; +import org.eclipse.rdf4j.sail.nativerdf.model.NativeValue; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.DynamicTest; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.TestFactory; +import org.junit.jupiter.api.io.TempDir; + +class ValueStoreWalCorruptRecoveryTest { + + private static final ValueFactory VF = SimpleValueFactory.getInstance(); + + @TempDir + Path tempDir; + + @BeforeEach + void setUp() { + NativeStore.SOFT_FAIL_ON_CORRUPT_DATA_AND_REPAIR_INDEXES = true; + } + + @AfterEach + void tearDown() { + NativeStore.SOFT_FAIL_ON_CORRUPT_DATA_AND_REPAIR_INDEXES = false; + } + + @Test + void corruptValueIsRecoveredFromWal() throws Exception { + Path walDir = tempDir.resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + Files.createDirectories(walDir); + ValueStoreWalConfig config = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid(UUID.randomUUID().toString()) + .build(); + + File valueDir = tempDir.resolve("values").toFile(); + Files.createDirectories(valueDir.toPath()); + + String label = "recover-me"; + int id; + try (ValueStoreWAL wal = ValueStoreWAL.open(config)) { + try (ValueStore store = new ValueStore(valueDir, false, + ValueStore.VALUE_CACHE_SIZE, ValueStore.VALUE_ID_CACHE_SIZE, + ValueStore.NAMESPACE_CACHE_SIZE, ValueStore.NAMESPACE_ID_CACHE_SIZE, wal)) { + Literal lit = VF.createLiteral(label); + id = store.storeValue(lit); + var lsn = store.drainPendingWalHighWaterMark(); + assertThat(lsn).isPresent(); + wal.awaitDurable(lsn.getAsLong()); + } + } + + // Corrupt the first byte (type marker) of the value record in values.dat for this id + File idFile = new File(valueDir, "values.id"); + File datFile = new File(valueDir, "values.dat"); + try (IDFile ids = new IDFile(idFile)) { + long offset = ids.getOffset(id); + try (RandomAccessFile raf = new RandomAccessFile(datFile, "rw")) { + // overwrite length to 0 to trigger empty data array corruption path + raf.seek(offset); + raf.writeInt(0); + } + } + + // Reopen store with WAL enabled and retrieve the value; it should be a CorruptValue with a recovered value + // attached + try (ValueStore store = new ValueStore(valueDir, false, + ValueStore.VALUE_CACHE_SIZE, ValueStore.VALUE_ID_CACHE_SIZE, + ValueStore.NAMESPACE_CACHE_SIZE, ValueStore.NAMESPACE_ID_CACHE_SIZE, + ValueStoreWAL.open(config))) { + NativeValue v = store.getValue(id); + assertThat(v.stringValue()).isEqualTo(label); + } + } + + @Test + void autoRecoversMissingValueFilesOnOpen() throws Exception { + Path walDir = tempDir.resolve("wal-auto-recover"); + Files.createDirectories(walDir); + ValueStoreWalConfig config = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid(UUID.randomUUID().toString()) + .recoverValueStoreOnOpen(true) + .build(); + + File valueDir = tempDir.resolve("values-auto").toFile(); + Files.createDirectories(valueDir.toPath()); + + String label = "auto-recover"; + int id; + try (ValueStoreWAL wal = ValueStoreWAL.open(config); + ValueStore store = new ValueStore(valueDir, false, ValueStore.VALUE_CACHE_SIZE, + ValueStore.VALUE_ID_CACHE_SIZE, ValueStore.NAMESPACE_CACHE_SIZE, + ValueStore.NAMESPACE_ID_CACHE_SIZE, wal)) { + id = store.storeValue(VF.createLiteral(label)); + OptionalLong pending = store.drainPendingWalHighWaterMark(); + assertThat(pending).isPresent(); + wal.awaitDurable(pending.getAsLong()); + } + + Files.deleteIfExists(valueDir.toPath().resolve("values.dat")); + Files.deleteIfExists(valueDir.toPath().resolve("values.id")); + Files.deleteIfExists(valueDir.toPath().resolve("values.hash")); + + try (ValueStoreWAL wal = ValueStoreWAL.open(config); + ValueStore store = new ValueStore(valueDir, false, ValueStore.VALUE_CACHE_SIZE, + ValueStore.VALUE_ID_CACHE_SIZE, ValueStore.NAMESPACE_CACHE_SIZE, + ValueStore.NAMESPACE_ID_CACHE_SIZE, wal)) { + NativeValue value = store.getValue(id); + assertThat(value).isNotNull(); + assertThat(value.stringValue()).isEqualTo(label); + } + } + + @Test + void autoRecoversMissingInteriorValueFromWal() throws Exception { + Path walDir = tempDir.resolve("wal-auto-recover-mid"); + Files.createDirectories(walDir); + ValueStoreWalConfig config = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid(UUID.randomUUID().toString()) + .recoverValueStoreOnOpen(true) + .build(); + + File valueDir = tempDir.resolve("values-auto-mid").toFile(); + Files.createDirectories(valueDir.toPath()); + + int targetIndex = 50; + String targetLabel = "auto-recover-mid-" + targetIndex; + int targetId = -1; + try (ValueStoreWAL wal = ValueStoreWAL.open(config); + ValueStore store = new ValueStore(valueDir, false, ValueStore.VALUE_CACHE_SIZE, + ValueStore.VALUE_ID_CACHE_SIZE, ValueStore.NAMESPACE_CACHE_SIZE, + ValueStore.NAMESPACE_ID_CACHE_SIZE, wal)) { + for (int i = 0; i < 100; i++) { + int id = store.storeValue(VF.createLiteral("auto-recover-mid-" + i)); + if (i == targetIndex) { + targetId = id; + } + } + var pending = store.drainPendingWalHighWaterMark(); + assertThat(pending).isPresent(); + wal.awaitDurable(pending.getAsLong()); + } + assertThat(targetId).isGreaterThan(0); + + try (IDFile ids = new IDFile(new File(valueDir, "values.id")); + RandomAccessFile raf = new RandomAccessFile(new File(valueDir, "values.dat"), "rw")) { + long offset = ids.getOffset(targetId); + raf.seek(offset); + raf.writeInt(0); + } + + try (ValueStoreWAL wal = ValueStoreWAL.open(config); + ValueStore store = new ValueStore(valueDir, false, ValueStore.VALUE_CACHE_SIZE, + ValueStore.VALUE_ID_CACHE_SIZE, ValueStore.NAMESPACE_CACHE_SIZE, + ValueStore.NAMESPACE_ID_CACHE_SIZE, wal)) { + NativeValue value = store.getValue(targetId); + assertThat(value).isInstanceOf(NativeLiteral.class); + assertThat(value.stringValue()).isEqualTo(targetLabel); + } + } + + @Test + void recoversValueWhenIdEntryPointsInsideRecord() throws Exception { + Path walDir = tempDir.resolve("wal-id-entry"); + Files.createDirectories(walDir); + ValueStoreWalConfig config = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid(UUID.randomUUID().toString()) + .build(); + + File valueDir = tempDir.resolve("values-id-entry").toFile(); + Files.createDirectories(valueDir.toPath()); + + String label = "id-entry-should-recover"; + int literalId; + try (ValueStoreWAL wal = ValueStoreWAL.open(config); + ValueStore store = new ValueStore(valueDir, false, ValueStore.VALUE_CACHE_SIZE, + ValueStore.VALUE_ID_CACHE_SIZE, ValueStore.NAMESPACE_CACHE_SIZE, + ValueStore.NAMESPACE_ID_CACHE_SIZE, wal)) { + literalId = store.storeValue(VF.createLiteral(label)); + OptionalLong pending = store.drainPendingWalHighWaterMark(); + assertThat(pending).isPresent(); + wal.awaitDurable(pending.getAsLong()); + } + + try (IDFile ids = new IDFile(new File(valueDir, "values.id"))) { + long currentOffset = ids.getOffset(literalId); + assertThat(currentOffset).isGreaterThan(0L); + ids.setOffset(literalId, currentOffset + 1); // point inside the literal record to corrupt the entry + } + + try (ValueStoreWAL wal = ValueStoreWAL.open(config); + ValueStore store = new ValueStore(valueDir, false, ValueStore.VALUE_CACHE_SIZE, + ValueStore.VALUE_ID_CACHE_SIZE, ValueStore.NAMESPACE_CACHE_SIZE, + ValueStore.NAMESPACE_ID_CACHE_SIZE, wal)) { + NativeValue recovered = store.getValue(literalId); + assertThat(recovered).isInstanceOf(NativeLiteral.class); + assertThat(recovered.stringValue()).isEqualTo(label); + } + } + + @Test + void corruptIriIsRecoveredFromWal() throws Exception { + Path walDir = tempDir.resolve("wal2"); + Files.createDirectories(walDir); + ValueStoreWalConfig config = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid(UUID.randomUUID().toString()) + .build(); + + File valueDir = tempDir.resolve("values2").toFile(); + Files.createDirectories(valueDir.toPath()); + + String iri = "http://ex.com/iri"; + int id; + try (ValueStoreWAL wal = ValueStoreWAL.open(config)) { + try (ValueStore store = new ValueStore(valueDir, false, ValueStore.VALUE_CACHE_SIZE, + ValueStore.VALUE_ID_CACHE_SIZE, ValueStore.NAMESPACE_CACHE_SIZE, ValueStore.NAMESPACE_ID_CACHE_SIZE, + wal)) { + id = store.storeValue(VF.createIRI(iri)); + var lsn = store.drainPendingWalHighWaterMark(); + assertThat(lsn).isPresent(); + wal.awaitDurable(lsn.getAsLong()); + } + } + + // corrupt entry length + File idFile = new File(valueDir, "values.id"); + File datFile = new File(valueDir, "values.dat"); + try (IDFile ids = new IDFile(idFile)) { + long offset = ids.getOffset(id); + try (RandomAccessFile raf = new RandomAccessFile(datFile, "rw")) { + raf.seek(offset); + raf.writeInt(0); + } + } + + try (ValueStore store = new ValueStore(valueDir, false, ValueStore.VALUE_CACHE_SIZE, + ValueStore.VALUE_ID_CACHE_SIZE, + ValueStore.NAMESPACE_CACHE_SIZE, ValueStore.NAMESPACE_ID_CACHE_SIZE, ValueStoreWAL.open(config))) { + NativeValue v = store.getValue(id); + assertThat(v.toString()).isEqualTo(iri); + + } + } + + @Test + void corruptBNodeIsRecoveredFromWal() throws Exception { + Path walDir = tempDir.resolve("wal3"); + Files.createDirectories(walDir); + ValueStoreWalConfig config = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid(UUID.randomUUID().toString()) + .build(); + + File valueDir = tempDir.resolve("values3").toFile(); + Files.createDirectories(valueDir.toPath()); + + String bnodeId = "bob"; + int id; + try (ValueStoreWAL wal = ValueStoreWAL.open(config)) { + try (ValueStore store = new ValueStore(valueDir, false, ValueStore.VALUE_CACHE_SIZE, + ValueStore.VALUE_ID_CACHE_SIZE, ValueStore.NAMESPACE_CACHE_SIZE, ValueStore.NAMESPACE_ID_CACHE_SIZE, + wal)) { + id = store.storeValue(VF.createBNode(bnodeId)); + var lsn = store.drainPendingWalHighWaterMark(); + assertThat(lsn).isPresent(); + wal.awaitDurable(lsn.getAsLong()); + } + } + + File idFile = new File(valueDir, "values.id"); + File datFile = new File(valueDir, "values.dat"); + try (IDFile ids = new IDFile(idFile)) { + long offset = ids.getOffset(id); + try (RandomAccessFile raf = new RandomAccessFile(datFile, "rw")) { + raf.seek(offset); + raf.writeInt(0); + } + } + + try (ValueStore store = new ValueStore(valueDir, false, ValueStore.VALUE_CACHE_SIZE, + ValueStore.VALUE_ID_CACHE_SIZE, + ValueStore.NAMESPACE_CACHE_SIZE, ValueStore.NAMESPACE_ID_CACHE_SIZE, ValueStoreWAL.open(config))) { + NativeValue v = store.getValue(id); + assertThat(v.stringValue()).isEqualTo(bnodeId); + } + } + + @TestFactory + Stream corruptAllLiteralTypesAreRecoveredFromWal() { + return provideLiterals().map(lit -> DynamicTest.dynamicTest( + "Recover literal: " + lit.toString(), + () -> runCorruptAndRecoverLiteralTest(lit) + )); + } + + private Stream provideLiterals() { + // Build a representative set covering all ValueFactory#createLiteral overloads supported here + var dt = VF.createIRI("http://example.com/dt"); + + XMLGregorianCalendar xmlCal; + try { + xmlCal = DatatypeFactory.newInstance().newXMLGregorianCalendar("2020-01-02T03:04:05Z"); + } catch (Exception e) { + throw new RuntimeException(e); + } + + return Stream.of( + // String + VF.createLiteral("simple-string"), + VF.createLiteral("hello", "en"), + VF.createLiteral("42", dt), + VF.createLiteral("123", CoreDatatype.XSD.INTEGER), + VF.createLiteral("abc", dt, CoreDatatype.NONE), + + // Booleans and numerics + VF.createLiteral(true), + VF.createLiteral(false), + VF.createLiteral((byte) 7), + VF.createLiteral((short) 12), + VF.createLiteral(34), + VF.createLiteral(56L), + VF.createLiteral(56L, CoreDatatype.XSD.LONG), + VF.createLiteral(1.5f), + VF.createLiteral(2.5d), + VF.createLiteral(new BigInteger("789")), + VF.createLiteral(new BigDecimal("123.456")), + + // TemporalAccessor and TemporalAmount + VF.createLiteral(LocalDate.of(2020, 1, 2)), + VF.createLiteral(LocalTime.of(3, 4, 5, 123_000_000)), + VF.createLiteral(LocalDateTime.of(2020, 1, 2, 3, 4, 5, 123_000_000)), + VF.createLiteral(OffsetDateTime.of(2020, 1, 2, 3, 4, 5, 0, ZoneOffset.UTC)), + VF.createLiteral(Period.of(1, 2, 3)), + VF.createLiteral(Duration.ofHours(5).plusMinutes(6).plusSeconds(7)), + + // XMLGregorianCalendar and Date + VF.createLiteral(xmlCal), + VF.createLiteral(new Date(1_577_926_245_000L)) // 2020-01-02T03:04:05Z + ); + } + + private void runCorruptAndRecoverLiteralTest(Literal lit) throws Exception { + Path walDir = tempDir.resolve("wal-lit-" + UUID.randomUUID()) + .resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + Files.createDirectories(walDir); + ValueStoreWalConfig config = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid(UUID.randomUUID().toString()) + .build(); + + File valueDir = walDir.getParent().resolve("values").toFile(); + Files.createDirectories(valueDir.toPath()); + + int id; + try (ValueStoreWAL wal = ValueStoreWAL.open(config)) { + try (ValueStore store = new ValueStore(valueDir, false, + ValueStore.VALUE_CACHE_SIZE, ValueStore.VALUE_ID_CACHE_SIZE, + ValueStore.NAMESPACE_CACHE_SIZE, ValueStore.NAMESPACE_ID_CACHE_SIZE, wal)) { + id = store.storeValue(lit); + var lsn = store.drainPendingWalHighWaterMark(); + assertThat(lsn).isPresent(); + wal.awaitDurable(lsn.getAsLong()); + } + } + + // Corrupt the value record length to trigger recovery path + File idFile = new File(valueDir, "values.id"); + File datFile = new File(valueDir, "values.dat"); + try (IDFile ids = new IDFile(idFile)) { + long offset = ids.getOffset(id); + try (RandomAccessFile raf = new RandomAccessFile(datFile, "rw")) { + raf.seek(offset); + raf.writeInt(0); + } + } + + // Reopen and verify recovered string label equals original + try (ValueStore store = new ValueStore(valueDir, false, + ValueStore.VALUE_CACHE_SIZE, ValueStore.VALUE_ID_CACHE_SIZE, + ValueStore.NAMESPACE_CACHE_SIZE, ValueStore.NAMESPACE_ID_CACHE_SIZE, + ValueStoreWAL.open(config))) { + NativeValue v = store.getValue(id); + assertThat(v.stringValue()).isEqualTo(lit.stringValue()); + } + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalDeletionDuringWriteTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalDeletionDuringWriteTest.java new file mode 100644 index 00000000000..789e6e291b4 --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalDeletionDuringWriteTest.java @@ -0,0 +1,123 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ + +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.Comparator; +import java.util.List; +import java.util.OptionalLong; +import java.util.UUID; +import java.util.regex.Matcher; +import java.util.regex.Pattern; + +import org.eclipse.rdf4j.model.IRI; +import org.eclipse.rdf4j.model.impl.SimpleValueFactory; +import org.eclipse.rdf4j.sail.nativerdf.ValueStore; +import org.eclipse.rdf4j.sail.nativerdf.datastore.DataStore; +import org.junit.jupiter.api.RepeatedTest; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +class ValueStoreWalDeletionDuringWriteTest { + + private static final Pattern SEGMENT_PATTERN = Pattern.compile("wal-(\\d{8})\\.v1"); + + @TempDir + Path tempDir; + + @Test + void asyncWalContinuesAfterCurrentSegmentDeletion() throws Exception { + Path walDir = tempDir.resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + ValueStoreWalConfig config = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid(UUID.randomUUID().toString()) + .maxSegmentBytes(1 << 12) + .syncPolicy(ValueStoreWalConfig.SyncPolicy.COMMIT) + .build(); + + Path valuesDir = tempDir.resolve("values"); + Files.createDirectories(valuesDir); + + List beforeDeletion = new ArrayList<>(); + List afterDeletion = new ArrayList<>(); + try (ValueStoreWAL wal = ValueStoreWAL.open(config); + ValueStore store = new ValueStore(valuesDir.toFile(), false, ValueStore.VALUE_CACHE_SIZE, + ValueStore.VALUE_ID_CACHE_SIZE, ValueStore.NAMESPACE_CACHE_SIZE, + ValueStore.NAMESPACE_ID_CACHE_SIZE, wal)) { + + for (int i = 0; i < 80; i++) { + beforeDeletion.add(mintUniqueIri(store, "before-" + i)); + } + drainAndAwait(store); + + Path currentSegment = locateCurrentSegment(walDir); + assertThat(currentSegment).as("current WAL segment").isNotNull(); + Files.deleteIfExists(currentSegment); + + for (int i = 80; i < 160; i++) { + afterDeletion.add(mintUniqueIri(store, "after-" + i)); + } + drainAndAwait(store); + } + + try (ValueStoreWalReader reader = ValueStoreWalReader.open(config); + DataStore ds = new DataStore(valuesDir.toFile(), "values")) { + ValueStoreWalRecovery recovery = new ValueStoreWalRecovery(); + var dictionary = recovery.replay(reader); + assertThat(afterDeletion).isNotEmpty(); + assertThat(dictionary.keySet()).as("WAL should retain post-deletion ids") + .containsAll(afterDeletion); + for (Integer id : beforeDeletion) { + assertThat(ds.getData(id)).as("ValueStore data should exist for id %s", id).isNotNull(); + } + for (Integer id : afterDeletion) { + assertThat(ds.getData(id)).as("ValueStore data should exist for id %s", id).isNotNull(); + } + } + } + + private static int mintUniqueIri(ValueStore store, String token) throws IOException { + IRI iri = SimpleValueFactory.getInstance().createIRI("http://example.com/value/" + token); + return store.storeValue(iri); + } + + private static void drainAndAwait(ValueStore store) throws IOException { + OptionalLong pending = store.drainPendingWalHighWaterMark(); + if (pending.isPresent()) { + store.awaitWalDurable(pending.getAsLong()); + } + } + + private static Path locateCurrentSegment(Path walDir) throws IOException { + if (!Files.isDirectory(walDir)) { + return null; + } + try (var stream = Files.list(walDir)) { + return stream.filter(path -> path.getFileName().toString().endsWith(".v1")) + .max(Comparator.comparingInt(ValueStoreWalDeletionDuringWriteTest::segmentSequence)) + .orElse(null); + } + } + + private static int segmentSequence(Path path) { + Matcher matcher = SEGMENT_PATTERN.matcher(path.getFileName().toString()); + if (!matcher.matches()) { + return -1; + } + return Integer.parseInt(matcher.group(1)); + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalDurabilityRecoveryTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalDurabilityRecoveryTest.java new file mode 100644 index 00000000000..6f61f91d905 --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalDurabilityRecoveryTest.java @@ -0,0 +1,163 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.File; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; + +import org.eclipse.rdf4j.common.io.ByteArrayUtil; +import org.eclipse.rdf4j.common.io.NioFile; +import org.eclipse.rdf4j.model.IRI; +import org.eclipse.rdf4j.model.ValueFactory; +import org.eclipse.rdf4j.model.impl.SimpleValueFactory; +import org.eclipse.rdf4j.model.util.Values; +import org.eclipse.rdf4j.repository.Repository; +import org.eclipse.rdf4j.repository.RepositoryConnection; +import org.eclipse.rdf4j.repository.sail.SailRepository; +import org.eclipse.rdf4j.sail.nativerdf.NativeStore; +import org.eclipse.rdf4j.sail.nativerdf.testutil.FailureInjectingFileChannel; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +/** + * Proves that a NativeStore with forceSync disabled can be fully recovered from a WAL that runs with SyncPolicy.COMMIT + * and synchronous bootstrap on open ensuring durability before commit returns. + */ +class ValueStoreWalDurabilityRecoveryTest { + + @TempDir + Path tempDir; + + private static final ValueFactory VF = SimpleValueFactory.getInstance(); + + @Test + void recoversFromLostValueStoreUsingWALCommitDurability() throws Exception { + // Install a delegating FileChannel factory (no failures by default), proving injection works + NioFile.setChannelFactoryForTesting( + (path, options) -> new FailureInjectingFileChannel(java.nio.channels.FileChannel.open(path, options))); + + File dataDir = tempDir.resolve("store").toFile(); + dataDir.mkdirs(); + + NativeStore store = new NativeStore(dataDir, "spoc,posc"); + store.setForceSync(false); // ValueStore won't fsync + store.setWalSyncPolicy(ValueStoreWalConfig.SyncPolicy.COMMIT); // WAL fsyncs on commit + store.setWalSyncBootstrapOnOpen(false); + Repository repo = new SailRepository(store); + repo.init(); + + IRI p = VF.createIRI("http://ex/p"); + IRI s = VF.createIRI("http://ex/s"); + IRI o = VF.createIRI("http://ex/o"); + try (RepositoryConnection conn = repo.getConnection()) { + conn.begin(); + conn.add(s, p, o, Values.iri("urn:g")); + conn.commit(); // WAL should force+persist before this returns + } + repo.shutDown(); + + // Simulate crash that loses the ValueStore by deleting the value files, WAL remains + Files.deleteIfExists(dataDir.toPath().resolve("values.dat")); + Files.deleteIfExists(dataDir.toPath().resolve("values.id")); + Files.deleteIfExists(dataDir.toPath().resolve("values.hash")); + + // Manually recover the ValueStore from WAL to simulate crash recovery + Path walDir = dataDir.toPath().resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + String storeUuid = Files.readString(walDir.resolve("store.uuid"), StandardCharsets.UTF_8).trim(); + ValueStoreWalConfig cfg = ValueStoreWalConfig.builder().walDirectory(walDir).storeUuid(storeUuid).build(); + java.util.Map dictionary; + try (ValueStoreWalReader reader = ValueStoreWalReader.open(cfg)) { + ValueStoreWalRecovery recovery = new ValueStoreWalRecovery(); + dictionary = new java.util.LinkedHashMap<>(recovery.replay(reader)); + } + try (org.eclipse.rdf4j.sail.nativerdf.datastore.DataStore ds = new org.eclipse.rdf4j.sail.nativerdf.datastore.DataStore( + dataDir, "values", false)) { + for (ValueStoreWalRecord record : dictionary.values()) { + switch (record.valueKind()) { + case NAMESPACE: + ds.storeData(record.lexical().getBytes(StandardCharsets.UTF_8)); + break; + case IRI: + ds.storeData(encodeIri(record.lexical(), ds)); + break; + case BNODE: + byte[] idData = record.lexical().getBytes(StandardCharsets.UTF_8); + byte[] bnode = new byte[1 + idData.length]; + bnode[0] = 0x2; + ByteArrayUtil.put(idData, bnode, 1); + ds.storeData(bnode); + break; + default: + ds.storeData(encodeLiteral(record.lexical(), record.datatype(), record.language(), ds)); + break; + } + } + ds.sync(); + } + + // Restart store and verify statement is readable (dictionary present) + NativeStore store2 = new NativeStore(dataDir, "spoc,posc"); + store2.setForceSync(false); + store2.setWalSyncPolicy(ValueStoreWalConfig.SyncPolicy.COMMIT); + store2.setWalSyncBootstrapOnOpen(true); + Repository repo2 = new SailRepository(store2); + repo2.init(); + try (RepositoryConnection conn = repo2.getConnection()) { + long count = conn.getStatements(s, p, o, false, Values.iri("urn:g")).stream().count(); + assertThat(count).isEqualTo(1L); + } + repo2.shutDown(); + + // Remove factory to avoid impacting other tests + NioFile.setChannelFactoryForTesting(null); + } + + private byte[] encodeIri(String lexical, org.eclipse.rdf4j.sail.nativerdf.datastore.DataStore ds) throws Exception { + IRI iri = VF.createIRI(lexical); + String ns = iri.getNamespace(); + String local = iri.getLocalName(); + int nsId = ds.getID(ns.getBytes(StandardCharsets.UTF_8)); + if (nsId == -1) { + nsId = ds.storeData(ns.getBytes(StandardCharsets.UTF_8)); + } + byte[] localBytes = local.getBytes(StandardCharsets.UTF_8); + byte[] data = new byte[1 + 4 + localBytes.length]; + data[0] = 0x1; + ByteArrayUtil.putInt(nsId, data, 1); + ByteArrayUtil.put(localBytes, data, 5); + return data; + } + + private byte[] encodeLiteral(String label, String datatype, String language, + org.eclipse.rdf4j.sail.nativerdf.datastore.DataStore ds) throws Exception { + int dtId = -1; // -1 denotes UNKNOWN_ID + if (datatype != null && !datatype.isEmpty()) { + byte[] dtBytes = encodeIri(datatype, ds); + int id = ds.getID(dtBytes); + dtId = id == -1 ? ds.storeData(dtBytes) : id; + } + byte[] langBytes = language == null ? new byte[0] : language.getBytes(StandardCharsets.UTF_8); + byte[] labelBytes = label.getBytes(StandardCharsets.UTF_8); + byte[] data = new byte[1 + 4 + 1 + langBytes.length + labelBytes.length]; + data[0] = 0x3; + ByteArrayUtil.putInt(dtId, data, 1); + data[5] = (byte) (langBytes.length & 0xFF); + if (langBytes.length > 0) { + ByteArrayUtil.put(langBytes, data, 6); + } + ByteArrayUtil.put(labelBytes, data, 6 + langBytes.length); + return data; + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalForceWithoutWritesTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalForceWithoutWritesTest.java new file mode 100644 index 00000000000..8bdf84c3c33 --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalForceWithoutWritesTest.java @@ -0,0 +1,193 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.MappedByteBuffer; +import java.nio.channels.FileChannel; +import java.nio.channels.FileLock; +import java.nio.channels.ReadableByteChannel; +import java.nio.channels.WritableByteChannel; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.OptionalLong; +import java.util.UUID; + +import org.eclipse.rdf4j.model.Literal; +import org.eclipse.rdf4j.model.ValueFactory; +import org.eclipse.rdf4j.model.impl.SimpleValueFactory; +import org.eclipse.rdf4j.sail.nativerdf.ValueStore; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +class ValueStoreWalForceWithoutWritesTest { + + private static final ValueFactory VF = SimpleValueFactory.getInstance(); + + @TempDir + Path tempDir; + + @AfterEach + void resetChannelFactory() { + ValueStoreWAL.resetChannelOpenerForTesting(); + } + + @Test + void doesNotForceFreshChannels() throws Exception { + List violations = Collections.synchronizedList(new ArrayList<>()); + ValueStoreWAL.setChannelOpenerForTesting((path, options) -> new TrackingFileChannel( + FileChannel.open(path, options), path, violations)); + + Path walDir = tempDir.resolve("wal"); + Files.createDirectories(walDir); + ValueStoreWalConfig config = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid(UUID.randomUUID().toString()) + .maxSegmentBytes(2 * 1024) + .batchBufferBytes(8 * 1024) + .syncPolicy(ValueStoreWalConfig.SyncPolicy.INTERVAL) + .build(); + + Path valuesDir = tempDir.resolve("values"); + Files.createDirectories(valuesDir); + + try (ValueStoreWAL wal = ValueStoreWAL.open(config); + ValueStore store = new ValueStore(valuesDir.toFile(), false, ValueStore.VALUE_CACHE_SIZE, + ValueStore.VALUE_ID_CACHE_SIZE, ValueStore.NAMESPACE_CACHE_SIZE, + ValueStore.NAMESPACE_ID_CACHE_SIZE, wal)) { + Literal literal = VF.createLiteral("value-" + "x".repeat(8192)); + store.storeValue(literal); + OptionalLong pending = store.drainPendingWalHighWaterMark(); + assertThat(pending).isPresent(); + store.awaitWalDurable(pending.getAsLong()); + } + + assertThat(violations) + .as("force() must only occur on channels that performed writes") + .isEmpty(); + } + + private static final class TrackingFileChannel extends FileChannel { + private final FileChannel delegate; + private final Path path; + private final List violations; + private long bytesWritten; + + private TrackingFileChannel(FileChannel delegate, Path path, List violations) { + this.delegate = delegate; + this.path = path; + this.violations = violations; + } + + @Override + public int read(ByteBuffer dst) throws IOException { + return delegate.read(dst); + } + + @Override + public long read(ByteBuffer[] dsts, int offset, int length) throws IOException { + return delegate.read(dsts, offset, length); + } + + @Override + public int write(ByteBuffer src) throws IOException { + int written = delegate.write(src); + bytesWritten += Math.max(0, written); + return written; + } + + @Override + public long write(ByteBuffer[] srcs, int offset, int length) throws IOException { + long written = delegate.write(srcs, offset, length); + bytesWritten += Math.max(0, written); + return written; + } + + @Override + public long position() throws IOException { + return delegate.position(); + } + + @Override + public FileChannel position(long newPosition) throws IOException { + delegate.position(newPosition); + return this; + } + + @Override + public long size() throws IOException { + return delegate.size(); + } + + @Override + public FileChannel truncate(long size) throws IOException { + delegate.truncate(size); + return this; + } + + @Override + public void force(boolean metaData) throws IOException { + if (bytesWritten == 0) { + violations.add(path); + } + delegate.force(metaData); + } + + @Override + public long transferTo(long position, long count, WritableByteChannel target) throws IOException { + return delegate.transferTo(position, count, target); + } + + @Override + public long transferFrom(ReadableByteChannel src, long position, long count) throws IOException { + return delegate.transferFrom(src, position, count); + } + + @Override + public int read(ByteBuffer dst, long position) throws IOException { + return delegate.read(dst, position); + } + + @Override + public int write(ByteBuffer src, long position) throws IOException { + int written = delegate.write(src, position); + bytesWritten += Math.max(0, written); + return written; + } + + @Override + protected void implCloseChannel() throws IOException { + delegate.close(); + } + + @Override + public FileLock lock(long position, long size, boolean shared) throws IOException { + return delegate.lock(position, size, shared); + } + + @Override + public FileLock tryLock(long position, long size, boolean shared) throws IOException { + return delegate.tryLock(position, size, shared); + } + + @Override + public MappedByteBuffer map(MapMode mode, long position, long size) throws IOException { + return delegate.map(mode, position, size); + } + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalHashTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalHashTest.java new file mode 100644 index 00000000000..de46818ad4b --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalHashTest.java @@ -0,0 +1,104 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.File; +import java.nio.charset.StandardCharsets; +import java.nio.file.Path; +import java.util.UUID; +import java.util.zip.CRC32C; + +import org.eclipse.rdf4j.model.IRI; +import org.eclipse.rdf4j.model.impl.SimpleValueFactory; +import org.eclipse.rdf4j.sail.nativerdf.ValueStore; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.Timeout; +import org.junit.jupiter.api.io.TempDir; + +/** + * Reproduces incorrect WAL hash computation (CRC32C state reuse across calls). + */ +class ValueStoreWalHashTest { + + @TempDir + Path tempDir; + + @Test + @Timeout(10) + void walHashesMatchFreshCrcForEachRecord() throws Exception { + // Arrange: temp data dir and WAL config + Path dataDir = tempDir.resolve("store"); + Path walDir = tempDir.resolve("wal"); + dataDir.toFile().mkdirs(); + walDir.toFile().mkdirs(); + + String storeUuid = UUID.randomUUID().toString(); + ValueStoreWalConfig config = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid(storeUuid) + .build(); + ValueStoreWAL wal = ValueStoreWAL.open(config); + + try (ValueStore vs = new ValueStore(new File(dataDir.toString()), false, + ValueStore.VALUE_CACHE_SIZE, ValueStore.VALUE_ID_CACHE_SIZE, + ValueStore.NAMESPACE_CACHE_SIZE, ValueStore.NAMESPACE_ID_CACHE_SIZE, wal)) { + + IRI a = SimpleValueFactory.getInstance().createIRI("http://example.org/a"); + IRI b = SimpleValueFactory.getInstance().createIRI("http://example.org/b"); + + // Act: mint two values in sequence on the same thread + int idA = vs.storeValue(a); + int idB = vs.storeValue(b); + assertThat(idA).isGreaterThan(0); + assertThat(idB).isGreaterThan(0); + } + + // Ensure WAL is fully flushed and closed + wal.close(); + + // Assert: read back WAL and verify each record's hash equals a fresh CRC of its own fields + try (ValueStoreWalReader reader = ValueStoreWalReader.open(config)) { + var it = reader.iterator(); + int seen = 0; + while (it.hasNext()) { + ValueStoreWalRecord r = it.next(); + int expected = freshCrc32c(r.valueKind(), r.lexical(), r.datatype(), r.language()); + // This assertion will fail on the second record with the buggy implementation + assertThat(r.hash()) + .as("hash should equal CRC32C(kind,lex,dt,lang) for id=" + r.id()) + .isEqualTo(expected); + seen++; + } + assertThat(seen).isGreaterThanOrEqualTo(2); + } + } + + private static int freshCrc32c(ValueStoreWalValueKind kind, String lexical, String datatype, String language) { + CRC32C crc32c = new CRC32C(); + crc32c.update((byte) kind.code()); + update(crc32c, lexical); + crc32c.update((byte) 0); + update(crc32c, datatype); + crc32c.update((byte) 0); + update(crc32c, language); + return (int) crc32c.getValue(); + } + + private static void update(CRC32C crc32c, String value) { + if (value == null || value.isEmpty()) { + return; + } + byte[] bytes = value.getBytes(StandardCharsets.UTF_8); + crc32c.update(bytes, 0, bytes.length); + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalIntegrationTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalIntegrationTest.java new file mode 100644 index 00000000000..2a98fc9bbda --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalIntegrationTest.java @@ -0,0 +1,246 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.File; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Map; +import java.util.Objects; +import java.util.OptionalLong; +import java.util.UUID; +import java.util.concurrent.TimeUnit; + +import org.eclipse.rdf4j.model.IRI; +import org.eclipse.rdf4j.model.Literal; +import org.eclipse.rdf4j.model.ValueFactory; +import org.eclipse.rdf4j.model.impl.SimpleValueFactory; +import org.eclipse.rdf4j.model.vocabulary.XMLSchema; +import org.eclipse.rdf4j.sail.nativerdf.ValueStore; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +class ValueStoreWalIntegrationTest { + + private static final ValueFactory VF = SimpleValueFactory.getInstance(); + + @TempDir + Path tempDir; + + @Test + void purgeDropsQueuedFramesOnClear() throws Exception { + Path walDir = tempDir.resolve("wal-purge"); + Files.createDirectories(walDir); + ValueStoreWalConfig config = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid(UUID.randomUUID().toString()) + // Default COMMIT policy: do not auto-flush unless forced + .syncPolicy(ValueStoreWalConfig.SyncPolicy.COMMIT) + .build(); + + File valueDir = tempDir.resolve("values-purge").toFile(); + Files.createDirectories(valueDir.toPath()); + + // Enqueue a single value and immediately clear() the store, which purges the WAL. + try (ValueStoreWAL wal = ValueStoreWAL.open(config); + ValueStore store = new ValueStore(valueDir, false, ValueStore.VALUE_CACHE_SIZE, + ValueStore.VALUE_ID_CACHE_SIZE, ValueStore.NAMESPACE_CACHE_SIZE, + ValueStore.NAMESPACE_ID_CACHE_SIZE, wal)) { + store.storeValue(VF.createLiteral("to-be-dropped")); + // Intentionally do not awaitDurable: the record remains queued/in-memory + store.clear(); // triggers WAL purge + + // Now add a post-clear value and force durability. If the purge didn't drop queued frames, + // the pre-clear value will be flushed together with this post-clear record. + store.storeValue(VF.createLiteral("after-clear")); + var lsn = store.drainPendingWalHighWaterMark(); + if (lsn.isPresent()) { + wal.awaitDurable(lsn.getAsLong()); + } + } + + // Give the background writer a brief window to act after purge. + long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(2); + boolean hasMinted = false; + while (System.nanoTime() < deadline) { + try (ValueStoreWalReader reader = ValueStoreWalReader.open(config)) { + var scan = reader.scan(); + hasMinted = scan.records().stream().anyMatch(r -> "to-be-dropped".equals(r.lexical())); + } + if (hasMinted) { + break; // if bug exists, record may appear quickly + } + TimeUnit.MILLISECONDS.sleep(25); + } + + // After purge, no pre-clear minted value must be recoverable from the WAL. + assertThat(hasMinted).isFalse(); + } + + void logsMintedValueRecords() throws Exception { + Path walDir = tempDir.resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + Files.createDirectories(walDir); + ValueStoreWalConfig config = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid(UUID.randomUUID().toString()) + .build(); + + try (ValueStoreWAL wal = ValueStoreWAL.open(config)) { + File valueDir = tempDir.resolve("values").toFile(); + Files.createDirectories(valueDir.toPath()); + try (ValueStore store = new ValueStore(valueDir, false, ValueStore.VALUE_CACHE_SIZE, + ValueStore.VALUE_ID_CACHE_SIZE, ValueStore.NAMESPACE_CACHE_SIZE, + ValueStore.NAMESPACE_ID_CACHE_SIZE, wal)) { + Literal literal = VF.createLiteral("hello"); + store.storeValue(literal); + + OptionalLong lsn = store.drainPendingWalHighWaterMark(); + assertThat(lsn).isPresent(); + + wal.awaitDurable(lsn.getAsLong()); + } + + ValueStoreWalReader reader = ValueStoreWalReader.open(config); + ValueStoreWalReader.ScanResult scan = reader.scan(); + reader.close(); + + assertThat(scan.records()).hasSize(3); + assertThat(scan.records()) + .anyMatch(record -> record.valueKind() == ValueStoreWalValueKind.NAMESPACE + && record.lexical().equals(XMLSchema.NAMESPACE)); + assertThat(scan.records()) + .anyMatch(record -> record.valueKind() == ValueStoreWalValueKind.IRI + && record.lexical().equals(XMLSchema.STRING.stringValue())); + assertThat(scan.records()) + .anyMatch(record -> record.valueKind() == ValueStoreWalValueKind.LITERAL + && record.lexical().equals("hello") + && record.datatype().equals(XMLSchema.STRING.stringValue())); + } + } + + @Test + void recoveryRebuildsMintedEntries() throws Exception { + Path walDir = tempDir.resolve("wal2"); + Files.createDirectories(walDir); + ValueStoreWalConfig config = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid(UUID.randomUUID().toString()) + .build(); + + Literal literal = VF.createLiteral("world", "en"); + IRI datatype = VF.createIRI("http://example.com/datatype"); + + try (ValueStoreWAL wal = ValueStoreWAL.open(config)) { + File valueDir = tempDir.resolve("values2").toFile(); + Files.createDirectories(valueDir.toPath()); + try (ValueStore store = new ValueStore(valueDir, false, ValueStore.VALUE_CACHE_SIZE, + ValueStore.VALUE_ID_CACHE_SIZE, ValueStore.NAMESPACE_CACHE_SIZE, + ValueStore.NAMESPACE_ID_CACHE_SIZE, wal)) { + store.storeValue(literal); + store.storeValue(VF.createIRI("http://example.com/resource")); + store.storeValue(datatype); + OptionalLong lsn = store.drainPendingWalHighWaterMark(); + assertThat(lsn).isPresent(); + wal.awaitDurable(lsn.getAsLong()); + } + } + + try (ValueStoreWalReader reader = ValueStoreWalReader.open(config)) { + ValueStoreWalRecovery recovery = new ValueStoreWalRecovery(); + Map dictionary = recovery.replay(reader); + assertThat(dictionary).isNotEmpty(); + assertThat(dictionary.values()) + .anyMatch(record -> record.valueKind() == ValueStoreWalValueKind.LITERAL + && record.lexical().equals("world")); + assertThat(dictionary.values()) + .anyMatch(record -> record.valueKind() == ValueStoreWalValueKind.IRI + && record.lexical().equals("http://example.com/resource")); + } + } + + @Test + void enablingWalOnPopulatedStoreRebuildsExistingEntries() throws Exception { + Path valuesPath = tempDir.resolve("values-existing"); + Files.createDirectories(valuesPath); + File valueDir = valuesPath.toFile(); + + IRI existingIri = VF.createIRI("http://example.com/existing/one"); + Literal existingLiteral = VF.createLiteral("existing-literal", "en"); + + try (ValueStore store = new ValueStore(valueDir, false, ValueStore.VALUE_CACHE_SIZE, + ValueStore.VALUE_ID_CACHE_SIZE, ValueStore.NAMESPACE_CACHE_SIZE, + ValueStore.NAMESPACE_ID_CACHE_SIZE, null)) { + store.storeValue(existingIri); + store.storeValue(existingLiteral); + } + + Path walDir = tempDir.resolve("wal-existing"); + Files.createDirectories(walDir); + ValueStoreWalConfig config = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid(UUID.randomUUID().toString()) + .build(); + + IRI newIri = VF.createIRI("http://example.com/new"); + + try (ValueStoreWAL wal = ValueStoreWAL.open(config); + ValueStore store = new ValueStore(valueDir, false, ValueStore.VALUE_CACHE_SIZE, + ValueStore.VALUE_ID_CACHE_SIZE, ValueStore.NAMESPACE_CACHE_SIZE, + ValueStore.NAMESPACE_ID_CACHE_SIZE, wal)) { + + store.storeValue(newIri); + OptionalLong lsn = store.drainPendingWalHighWaterMark(); + if (lsn.isPresent()) { + wal.awaitDurable(lsn.getAsLong()); + } + + ValueStoreWalRecovery recovery = new ValueStoreWalRecovery(); + Map dictionary = Map.of(); + long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(5); + boolean hasExistingIri = false; + boolean hasExistingLiteral = false; + while (System.nanoTime() < deadline && (!hasExistingIri || !hasExistingLiteral)) { + try (ValueStoreWalReader reader = ValueStoreWalReader.open(config)) { + dictionary = recovery.replay(reader); + } + hasExistingIri = dictionary.values() + .stream() + .anyMatch(record -> record.valueKind() == ValueStoreWalValueKind.IRI + && record.lexical().equals(existingIri.stringValue())); + hasExistingLiteral = dictionary.values() + .stream() + .anyMatch(record -> record.valueKind() == ValueStoreWalValueKind.LITERAL + && record.lexical().equals(existingLiteral.getLabel()) + && Objects.toString(record.language(), "") + .equals(existingLiteral.getLanguage().orElse(""))); + if (!hasExistingIri || !hasExistingLiteral) { + TimeUnit.MILLISECONDS.sleep(25); + } + } + + assertThat(hasExistingIri).isTrue(); + assertThat(hasExistingLiteral).isTrue(); + assertThat(dictionary.values()) + .anyMatch(record -> record.valueKind() == ValueStoreWalValueKind.IRI + && record.lexical().equals(newIri.stringValue())); + } + + try (var stream = Files.list(walDir)) { + assertThat(stream + .filter(Files::isRegularFile) + .map(path -> path.getFileName().toString()) + .filter(name -> name.startsWith("wal-"))) + .allMatch(name -> name.matches("wal-[1-9]\\d*\\.v1(?:\\.gz)?")); + } + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalIntervalFsyncTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalIntervalFsyncTest.java new file mode 100644 index 00000000000..fe74d3cd079 --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalIntervalFsyncTest.java @@ -0,0 +1,121 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.junit.jupiter.api.Assertions.fail; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.time.Duration; +import java.util.ArrayList; +import java.util.List; +import java.util.OptionalLong; +import java.util.UUID; +import java.util.concurrent.TimeUnit; +import java.util.function.BooleanSupplier; + +import org.eclipse.rdf4j.model.Literal; +import org.eclipse.rdf4j.model.ValueFactory; +import org.eclipse.rdf4j.model.impl.SimpleValueFactory; +import org.eclipse.rdf4j.sail.nativerdf.ValueStore; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +class ValueStoreWalIntervalFsyncTest { + + private static final ValueFactory VF = SimpleValueFactory.getInstance(); + + @TempDir + Path tempDir; + + @AfterEach + void clearListener() { + ValueStoreWalDebug.clearForceListener(); + } + + @Test + void intervalForcesOnRotationAndCompression() throws Exception { + Path walDir = tempDir.resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + Files.createDirectories(walDir); + + List forced = new ArrayList<>(); + ValueStoreWalDebug.setForceListener(path -> { + synchronized (forced) { + forced.add(path); + } + }); + + ValueStoreWalConfig config = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid(UUID.randomUUID().toString()) + .maxSegmentBytes(2 * 1024) + .batchBufferBytes(8 * 1024) + .syncPolicy(ValueStoreWalConfig.SyncPolicy.INTERVAL) + .syncInterval(Duration.ofHours(1)) + .build(); + + Path valuesDir = tempDir.resolve("values"); + Files.createDirectories(valuesDir); + + try (ValueStoreWAL wal = ValueStoreWAL.open(config); + ValueStore store = new ValueStore(valuesDir.toFile(), false, ValueStore.VALUE_CACHE_SIZE, + ValueStore.VALUE_ID_CACHE_SIZE, ValueStore.NAMESPACE_CACHE_SIZE, + ValueStore.NAMESPACE_ID_CACHE_SIZE, wal)) { + + Literal literal = VF.createLiteral(repeat('x', 8_192)); + store.storeValue(literal); + OptionalLong pending = store.drainPendingWalHighWaterMark(); + assertThat(pending).isPresent(); + store.awaitWalDurable(pending.getAsLong()); + + waitFor(() -> containsFileWithSuffix(walDir, ".v1.gz")); + } + + waitFor(() -> containsForcedPath(forced, ".v1")); + waitFor(() -> containsForcedPath(forced, ".v1.gz")); + } + + private static boolean containsFileWithSuffix(Path dir, String suffix) { + try { + return Files.list(dir).anyMatch(path -> path.getFileName().toString().endsWith(suffix)); + } catch (IOException e) { + return false; + } + } + + private static boolean containsForcedPath(List forced, String suffix) { + synchronized (forced) { + return forced.stream().anyMatch(path -> path.getFileName().toString().endsWith(suffix)); + } + } + + private static void waitFor(BooleanSupplier condition) throws InterruptedException { + long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(5); + while (System.nanoTime() < deadline) { + if (condition.getAsBoolean()) { + return; + } + Thread.sleep(10); + } + fail("condition not met before timeout"); + } + + private static String repeat(char ch, int count) { + StringBuilder builder = new StringBuilder(count); + for (int i = 0; i < count; i++) { + builder.append(ch); + } + return builder.toString(); + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalLargeRecordTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalLargeRecordTest.java new file mode 100644 index 00000000000..ed26dec28e0 --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalLargeRecordTest.java @@ -0,0 +1,117 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ + +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.File; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.OptionalLong; +import java.util.UUID; + +import org.eclipse.rdf4j.model.Literal; +import org.eclipse.rdf4j.model.impl.SimpleValueFactory; +import org.eclipse.rdf4j.sail.nativerdf.ValueStore; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +class ValueStoreWalLargeRecordTest { + + @TempDir + Path tempDir; + + @Test + void logsLargeLiteralExceedingBuffer() throws Exception { + // Create a WAL with default config (1 MiB batch buffer) + Path walDir = tempDir.resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + Files.createDirectories(walDir); + ValueStoreWalConfig config = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid(UUID.randomUUID().toString()) + .build(); + + // Build a ~128 MiB ASCII literal (bytes == chars) + int sizeBytes = 128 * 1024 * 1024; // 128 MiB + String large = "a".repeat(sizeBytes); + Literal largeLiteral = SimpleValueFactory.getInstance().createLiteral(large); + + try (ValueStoreWAL wal = ValueStoreWAL.open(config)) { + File valueDir = tempDir.resolve("values").toFile(); + Files.createDirectories(valueDir.toPath()); + try (ValueStore store = new ValueStore(valueDir, false, ValueStore.VALUE_CACHE_SIZE, + ValueStore.VALUE_ID_CACHE_SIZE, ValueStore.NAMESPACE_CACHE_SIZE, + ValueStore.NAMESPACE_ID_CACHE_SIZE, wal)) { + // Store the large literal and wait for durability + store.storeValue(largeLiteral); + OptionalLong lsn = store.drainPendingWalHighWaterMark(); + assertThat(lsn).isPresent(); + + // This currently fails due to BufferOverflowException in the writer thread + wal.awaitDurable(lsn.getAsLong()); + } + } + + // Sanity: ensure scan can see the record and its size matches + try (ValueStoreWalReader reader = ValueStoreWalReader.open(config)) { + ValueStoreWalReader.ScanResult scan = reader.scan(); + assertThat(scan.records()).anyMatch(r -> r.valueKind() == ValueStoreWalValueKind.LITERAL + && r.lexical().length() == sizeBytes); + } + } + + @Test + void logsLargeLiteralWithSmallSegmentLimit() throws Exception { + Path walDir = tempDir.resolve("wal-small"); + Files.createDirectories(walDir); + ValueStoreWalConfig config = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid(UUID.randomUUID().toString()) + .maxSegmentBytes(32 * 1024) + .build(); + + int sizeBytes = 50 * 1024; // 50 KiB > segment limit + String large = "b".repeat(sizeBytes); + Literal literal = SimpleValueFactory.getInstance().createLiteral(large); + + try (ValueStoreWAL wal = ValueStoreWAL.open(config)) { + File valueDir = tempDir.resolve("values-small").toFile(); + Files.createDirectories(valueDir.toPath()); + try (ValueStore store = new ValueStore(valueDir, false, ValueStore.VALUE_CACHE_SIZE, + ValueStore.VALUE_ID_CACHE_SIZE, ValueStore.NAMESPACE_CACHE_SIZE, + ValueStore.NAMESPACE_ID_CACHE_SIZE, wal)) { + store.storeValue(literal); + OptionalLong lsn = store.drainPendingWalHighWaterMark(); + assertThat(lsn).isPresent(); + wal.awaitDurable(lsn.getAsLong()); + } + } + + try (ValueStoreWalReader reader = ValueStoreWalReader.open(config)) { + ValueStoreWalReader.ScanResult scan = reader.scan(); + assertThat(scan.records()) + .anyMatch(r -> r.valueKind() == ValueStoreWalValueKind.LITERAL && r.lexical().equals(large)); + } + + ValueStoreWalSearch search = ValueStoreWalSearch.open(config); + ValueStoreWalValueKind[] foundKind = new ValueStoreWalValueKind[1]; + try (ValueStoreWalReader reader = ValueStoreWalReader.open(config)) { + for (ValueStoreWalRecord rec : reader.scan().records()) { + if (rec.lexical().equals(large)) { + foundKind[0] = rec.valueKind(); + break; + } + } + } + assertThat(foundKind[0]).isEqualTo(ValueStoreWalValueKind.LITERAL); + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReadSegmentSequenceTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReadSegmentSequenceTest.java new file mode 100644 index 00000000000..dbe43c36fd1 --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReadSegmentSequenceTest.java @@ -0,0 +1,84 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.UUID; +import java.util.zip.GZIPOutputStream; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import com.fasterxml.jackson.core.JsonFactory; +import com.fasterxml.jackson.core.JsonGenerator; + +class ValueStoreWalReadSegmentSequenceTest { + + @TempDir + Path tempDir; + + @Test + void readsSequenceFromUncompressed() throws Exception { + Path walDir = tempDir.resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + Files.createDirectories(walDir); + Path seg = walDir.resolve("wal-1.v1"); + Files.write(seg, buildHeaderFrame("store-" + UUID.randomUUID(), 42, 1)); + int seq = ValueStoreWAL.readSegmentSequence(seg); + assertThat(seq).isEqualTo(42); + } + + @Test + void readsSequenceFromCompressed() throws Exception { + Path walDir = tempDir.resolve("wal-gz"); + Files.createDirectories(walDir); + Path gz = walDir.resolve("wal-10.v1.gz"); + byte[] header = buildHeaderFrame("store-" + UUID.randomUUID(), 7, 10); + try (GZIPOutputStream gout = new GZIPOutputStream(Files.newOutputStream(gz))) { + gout.write(header); + gout.finish(); + } + int seq = ValueStoreWAL.readSegmentSequence(gz); + assertThat(seq).isEqualTo(7); + } + + private static byte[] buildHeaderFrame(String store, int segment, int firstId) throws IOException { + JsonFactory f = new JsonFactory(); + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + try (JsonGenerator g = f.createGenerator(baos)) { + g.writeStartObject(); + g.writeStringField("t", "V"); + g.writeNumberField("ver", 1); + g.writeStringField("store", store); + g.writeStringField("engine", "valuestore"); + g.writeNumberField("created", 0); + g.writeNumberField("segment", segment); + g.writeNumberField("firstId", firstId); + g.writeEndObject(); + } + baos.write('\n'); + byte[] json = baos.toByteArray(); + ByteBuffer buf = ByteBuffer.allocate(4 + json.length + 4).order(ByteOrder.LITTLE_ENDIAN); + buf.putInt(json.length); + buf.put(json); + buf.putInt(0); // CRC is ignored by readSegmentSequence + buf.flip(); + byte[] framed = new byte[buf.remaining()]; + buf.get(framed); + return framed; + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderGzipInvalidAndTruncatedTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderGzipInvalidAndTruncatedTest.java new file mode 100644 index 00000000000..330fdbccd43 --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderGzipInvalidAndTruncatedTest.java @@ -0,0 +1,137 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.zip.GZIPOutputStream; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import com.fasterxml.jackson.core.JsonFactory; +import com.fasterxml.jackson.core.JsonGenerator; + +/** + * Tests gzip path for invalid length and truncated CRC conditions. + */ +class ValueStoreWalReaderGzipInvalidAndTruncatedTest { + + @TempDir + Path tempDir; + + @Test + void invalidLengthMarksIncomplete() throws Exception { + Path walDir = tempDir.resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + Files.createDirectories(walDir); + Path gz = walDir.resolve("wal-1.v1.gz"); + try (GZIPOutputStream out = new GZIPOutputStream(Files.newOutputStream(gz))) { + // Write header frame correctly + frame(out, headerJson(1, 1)); + // Write an invalid frame length (0) and nothing else + ByteBuffer lb = ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putInt(0); + lb.flip(); + out.write(lb.array(), 0, 4); + out.finish(); + } + ValueStoreWalConfig cfg = ValueStoreWalConfig.builder().walDirectory(walDir).storeUuid("s").build(); + try (ValueStoreWalReader reader = ValueStoreWalReader.open(cfg)) { + ValueStoreWalReader.ScanResult res = reader.scan(); + assertThat(res.complete()).isFalse(); + assertThat(res.records()).isEmpty(); + } + } + + @Test + void truncatedCrcMarksIncomplete() throws Exception { + Path walDir = tempDir.resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + Files.createDirectories(walDir); + Path gz = walDir.resolve("wal-2.v1.gz"); + try (GZIPOutputStream out = new GZIPOutputStream(Files.newOutputStream(gz))) { + // Header frame + frame(out, headerJson(2, 1)); + // Minted frame with correct length and payload but omit CRC + byte[] json = mintedJson(1L, 1); + ByteBuffer lb = ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putInt(json.length); + lb.flip(); + out.write(lb.array(), 0, 4); + out.write(json); + // no CRC written -> truncated + out.finish(); + } + ValueStoreWalConfig cfg = ValueStoreWalConfig.builder().walDirectory(walDir).storeUuid("s").build(); + try (ValueStoreWalReader reader = ValueStoreWalReader.open(cfg)) { + ValueStoreWalReader.ScanResult res = reader.scan(); + assertThat(res.complete()).isFalse(); + assertThat(res.records()).isEmpty(); + } + } + + private static void frame(GZIPOutputStream out, byte[] json) throws IOException { + ByteBuffer lb = ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putInt(json.length); + lb.flip(); + out.write(lb.array(), 0, 4); + out.write(json); + int crc = crc32c(json); + ByteBuffer cb = ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putInt(crc); + cb.flip(); + out.write(cb.array(), 0, 4); + } + + private static int crc32c(byte[] data) { + java.util.zip.CRC32C c = new java.util.zip.CRC32C(); + c.update(data, 0, data.length); + return (int) c.getValue(); + } + + private static byte[] headerJson(int segment, int firstId) throws IOException { + JsonFactory f = new JsonFactory(); + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + try (JsonGenerator g = f.createGenerator(baos)) { + g.writeStartObject(); + g.writeStringField("t", "V"); + g.writeNumberField("ver", 1); + g.writeStringField("store", "s"); + g.writeStringField("engine", "valuestore"); + g.writeNumberField("created", 0); + g.writeNumberField("segment", segment); + g.writeNumberField("firstId", firstId); + g.writeEndObject(); + } + baos.write('\n'); + return baos.toByteArray(); + } + + private static byte[] mintedJson(long lsn, int id) throws IOException { + JsonFactory f = new JsonFactory(); + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + try (JsonGenerator g = f.createGenerator(baos)) { + g.writeStartObject(); + g.writeStringField("t", "M"); + g.writeNumberField("lsn", lsn); + g.writeNumberField("id", id); + g.writeStringField("vk", "I"); + g.writeStringField("lex", "http://ex/id" + id); + g.writeStringField("dt", ""); + g.writeStringField("lang", ""); + g.writeNumberField("hash", 0); + g.writeEndObject(); + } + baos.write('\n'); + return baos.toByteArray(); + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderHasSequenceGapsTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderHasSequenceGapsTest.java new file mode 100644 index 00000000000..da0e6132e2b --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderHasSequenceGapsTest.java @@ -0,0 +1,76 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.file.Files; +import java.nio.file.Path; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import com.fasterxml.jackson.core.JsonFactory; +import com.fasterxml.jackson.core.JsonGenerator; + +/** + * Ensures reader reports incomplete when segment sequences are non-contiguous (e.g., segments 1 and 3 present). + */ +class ValueStoreWalReaderHasSequenceGapsTest { + + @TempDir + Path tempDir; + + @Test + void sequenceGapsMarkIncomplete() throws Exception { + Path walDir = tempDir.resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + Files.createDirectories(walDir); + Files.write(walDir.resolve("wal-10.v1"), headerOnly(1, 10)); + Files.write(walDir.resolve("wal-20.v1"), headerOnly(3, 20)); + + ValueStoreWalConfig cfg = ValueStoreWalConfig.builder().walDirectory(walDir).storeUuid("s").build(); + try (ValueStoreWalReader reader = ValueStoreWalReader.open(cfg)) { + ValueStoreWalReader.ScanResult res = reader.scan(); + assertThat(res.records()).isEmpty(); + assertThat(res.complete()).isFalse(); + } + } + + private static byte[] headerOnly(int segment, int firstId) throws IOException { + JsonFactory f = new JsonFactory(); + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + try (JsonGenerator g = f.createGenerator(baos)) { + g.writeStartObject(); + g.writeStringField("t", "V"); + g.writeNumberField("ver", 1); + g.writeStringField("store", "s"); + g.writeStringField("engine", "valuestore"); + g.writeNumberField("created", 0); + g.writeNumberField("segment", segment); + g.writeNumberField("firstId", firstId); + g.writeEndObject(); + } + baos.write('\n'); + byte[] json = baos.toByteArray(); + ByteBuffer buf = ByteBuffer.allocate(4 + json.length + 4).order(ByteOrder.LITTLE_ENDIAN); + buf.putInt(json.length); + buf.put(json); + buf.putInt(0); + buf.flip(); + byte[] framed = new byte[buf.remaining()]; + buf.get(framed); + return framed; + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderInvalidFrameTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderInvalidFrameTest.java new file mode 100644 index 00000000000..efaead2e43c --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderInvalidFrameTest.java @@ -0,0 +1,87 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.UUID; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import com.fasterxml.jackson.core.JsonFactory; +import com.fasterxml.jackson.core.JsonGenerator; + +/** + * Ensures the reader marks the scan incomplete when encountering an invalid or oversized frame length. + */ +class ValueStoreWalReaderInvalidFrameTest { + + @TempDir + Path tempDir; + + @Test + void invalidLengthStopsScanAndMarksIncomplete() throws Exception { + Path walDir = tempDir.resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + Files.createDirectories(walDir); + + // Build an uncompressed segment with a valid header then an invalid next frame length (> MAX_FRAME_BYTES) + Path seg = walDir.resolve("wal-1.v1"); + byte[] header = headerFrame("s-" + UUID.randomUUID()); + ByteBuffer buf = ByteBuffer.allocate(header.length + 4).order(ByteOrder.LITTLE_ENDIAN); + buf.put(header); + buf.putInt(ValueStoreWAL.MAX_FRAME_BYTES + 1); // invalid length sentinel + Files.write(seg, buf.array()); + + ValueStoreWalConfig cfg = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid("x") + .build(); + try (ValueStoreWalReader reader = ValueStoreWalReader.open(cfg)) { + ValueStoreWalReader.ScanResult scan = reader.scan(); + assertThat(scan.complete()).isFalse(); + assertThat(scan.lastValidLsn()).isEqualTo(ValueStoreWAL.NO_LSN); + assertThat(scan.records()).isEmpty(); + } + } + + private static byte[] headerFrame(String store) throws IOException { + JsonFactory f = new JsonFactory(); + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + try (JsonGenerator g = f.createGenerator(baos)) { + g.writeStartObject(); + g.writeStringField("t", "V"); + g.writeNumberField("ver", 1); + g.writeStringField("store", store); + g.writeStringField("engine", "valuestore"); + g.writeNumberField("created", 0); + g.writeNumberField("segment", 1); + g.writeNumberField("firstId", 1); + g.writeEndObject(); + } + baos.write('\n'); + byte[] json = baos.toByteArray(); + ByteBuffer frame = ByteBuffer.allocate(4 + json.length + 4).order(ByteOrder.LITTLE_ENDIAN); + frame.putInt(json.length); + frame.put(json); + frame.putInt(0); + frame.flip(); + byte[] out = new byte[frame.remaining()]; + frame.get(out); + return out; + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderIteratorTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderIteratorTest.java new file mode 100644 index 00000000000..41b0a944267 --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderIteratorTest.java @@ -0,0 +1,91 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ + +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.Iterator; +import java.util.List; +import java.util.OptionalLong; +import java.util.UUID; + +import org.eclipse.rdf4j.model.impl.SimpleValueFactory; +import org.eclipse.rdf4j.sail.nativerdf.ValueStore; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +/** + * Tests for a streaming/iterator-style ValueStoreWalReader API that yields one record at a time in order. + */ +class ValueStoreWalReaderIteratorTest { + + @TempDir + Path tempDir; + + @Test + void iteratesRecordsInOrderAndMatchesScan() throws Exception { + Path walDir = tempDir.resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + Files.createDirectories(walDir); + ValueStoreWalConfig config = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid(UUID.randomUUID().toString()) + .build(); + + // Write a few values to generate WAL records + try (ValueStoreWAL wal = ValueStoreWAL.open(config)) { + Path valuesDir = tempDir.resolve("values"); + Files.createDirectories(valuesDir); + try (ValueStore store = new ValueStore( + valuesDir.toFile(), false, + ValueStore.VALUE_CACHE_SIZE, ValueStore.VALUE_ID_CACHE_SIZE, + ValueStore.NAMESPACE_CACHE_SIZE, ValueStore.NAMESPACE_ID_CACHE_SIZE, wal)) { + store.storeValue(SimpleValueFactory.getInstance().createLiteral("r1")); + store.storeValue(SimpleValueFactory.getInstance().createIRI("http://ex/r2")); + store.storeValue(SimpleValueFactory.getInstance().createLiteral("r3", "en")); + OptionalLong lsn = store.drainPendingWalHighWaterMark(); + assertThat(lsn).isPresent(); + wal.awaitDurable(lsn.getAsLong()); + } + } + + // Existing API for comparison + List scanned; + long lastValidLsn; + try (ValueStoreWalReader reader = ValueStoreWalReader.open(config)) { + ValueStoreWalReader.ScanResult res = reader.scan(); + scanned = res.records(); + lastValidLsn = res.lastValidLsn(); + } + + // New iterator API (to be implemented): iterate without preloading all + List iterated = new ArrayList<>(); + long iterLast = ValueStoreWAL.NO_LSN; + try (ValueStoreWalReader reader = ValueStoreWalReader.open(config)) { + Iterator it = reader.iterator(); // expected new API + while (it.hasNext()) { + ValueStoreWalRecord r = it.next(); + iterated.add(r); + if (r.lsn() > iterLast) { + iterLast = r.lsn(); + } + } + // After iteration, lastValidLsn() should reflect last good record + assertThat(reader.lastValidLsn()).isEqualTo(iterLast); + } + + assertThat(iterated).usingRecursiveComparison().isEqualTo(scanned); + assertThat(iterLast).isEqualTo(lastValidLsn); + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderJacksonTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderJacksonTest.java new file mode 100644 index 00000000000..64ab2dd7526 --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderJacksonTest.java @@ -0,0 +1,80 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ + +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.File; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.List; +import java.util.UUID; + +import org.eclipse.rdf4j.model.IRI; +import org.eclipse.rdf4j.model.Literal; +import org.eclipse.rdf4j.model.ValueFactory; +import org.eclipse.rdf4j.model.impl.SimpleValueFactory; +import org.eclipse.rdf4j.model.vocabulary.XMLSchema; +import org.eclipse.rdf4j.sail.nativerdf.ValueStore; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +class ValueStoreWalReaderJacksonTest { + + private static final ValueFactory VF = SimpleValueFactory.getInstance(); + + @TempDir + Path tempDir; + + @Test + void scanReturnsMintedRecordsWithEscapes() throws Exception { + Path walDir = tempDir.resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + Files.createDirectories(walDir); + ValueStoreWalConfig config = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid(UUID.randomUUID().toString()) + .build(); + + String specialText = "He said: \"Hello\\World\"\nNew line"; + try (ValueStoreWAL wal = ValueStoreWAL.open(config)) { + File valueDir = tempDir.resolve("values").toFile(); + Files.createDirectories(valueDir.toPath()); + try (ValueStore store = new ValueStore(valueDir, false, ValueStore.VALUE_CACHE_SIZE, + ValueStore.VALUE_ID_CACHE_SIZE, ValueStore.NAMESPACE_CACHE_SIZE, + ValueStore.NAMESPACE_ID_CACHE_SIZE, wal)) { + IRI iri = VF.createIRI("http://example.com/resource"); + Literal lit = VF.createLiteral(specialText, XMLSchema.STRING); + store.storeValue(iri); + store.storeValue(lit); + + var lsn = store.drainPendingWalHighWaterMark(); + assertThat(lsn).isPresent(); + wal.awaitDurable(lsn.getAsLong()); + } + } + + try (ValueStoreWalReader reader = ValueStoreWalReader.open(config)) { + ValueStoreWalReader.ScanResult scan = reader.scan(); + List records = scan.records(); + assertThat(records).isNotEmpty(); + assertThat(records.stream() + .anyMatch(r -> r.valueKind() == ValueStoreWalValueKind.IRI + && r.lexical().equals("http://example.com/resource"))) + .isTrue(); + assertThat(records.stream() + .anyMatch(r -> r.valueKind() == ValueStoreWalValueKind.LITERAL + && r.lexical().equals(specialText))) + .isTrue(); + assertThat(scan.lastValidLsn()).isGreaterThan(ValueStoreWAL.NO_LSN); + } + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderLastLsnNonMintedTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderLastLsnNonMintedTest.java new file mode 100644 index 00000000000..70aa76e8e39 --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderLastLsnNonMintedTest.java @@ -0,0 +1,128 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.List; +import java.util.zip.CRC32C; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import com.fasterxml.jackson.core.JsonFactory; +import com.fasterxml.jackson.core.JsonGenerator; + +/** + * Verifies that encountering non-minted frames (header 'V' and summary 'S') does not alter lastValidLsn; it should + * reflect the last minted record's LSN only. + */ +class ValueStoreWalReaderLastLsnNonMintedTest { + + @TempDir + Path tempDir; + + @Test + void lastValidLsnIgnoresNonMintedFrames() throws Exception { + Path walDir = tempDir.resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + Files.createDirectories(walDir); + Path seg = walDir.resolve("wal-1.v1"); + int mintedId = 10; + long mintedLsn = 42L; + ByteArrayOutputStream out = new ByteArrayOutputStream(); + frame(out, headerJson(1, 1)); + frame(out, mintedJson(mintedLsn, mintedId)); + frame(out, summaryJson(mintedId)); + Files.write(seg, out.toByteArray()); + + ValueStoreWalConfig cfg = ValueStoreWalConfig.builder().walDirectory(walDir).storeUuid("s").build(); + List recs = new ArrayList<>(); + try (ValueStoreWalReader reader = ValueStoreWalReader.open(cfg)) { + var it = reader.iterator(); + while (it.hasNext()) { + recs.add(it.next()); + } + assertThat(reader.lastValidLsn()).isEqualTo(mintedLsn); + } + assertThat(recs).hasSize(1); + assertThat(recs.get(0).id()).isEqualTo(mintedId); + } + + private static void frame(ByteArrayOutputStream out, byte[] json) { + ByteBuffer len = ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putInt(json.length); + len.flip(); + out.write(len.array(), 0, 4); + out.write(json, 0, json.length); + CRC32C c = new CRC32C(); + c.update(json, 0, json.length); + ByteBuffer crc = ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putInt((int) c.getValue()); + crc.flip(); + out.write(crc.array(), 0, 4); + } + + private static byte[] headerJson(int segment, int firstId) throws IOException { + JsonFactory f = new JsonFactory(); + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + try (JsonGenerator g = f.createGenerator(baos)) { + g.writeStartObject(); + g.writeStringField("t", "V"); + g.writeNumberField("ver", 1); + g.writeStringField("store", "s"); + g.writeStringField("engine", "valuestore"); + g.writeNumberField("created", 0); + g.writeNumberField("segment", segment); + g.writeNumberField("firstId", firstId); + g.writeEndObject(); + } + baos.write('\n'); + return baos.toByteArray(); + } + + private static byte[] mintedJson(long lsn, int id) throws IOException { + JsonFactory f = new JsonFactory(); + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + try (JsonGenerator g = f.createGenerator(baos)) { + g.writeStartObject(); + g.writeStringField("t", "M"); + g.writeNumberField("lsn", lsn); + g.writeNumberField("id", id); + g.writeStringField("vk", "I"); + g.writeStringField("lex", "http://ex/id" + id); + g.writeStringField("dt", ""); + g.writeStringField("lang", ""); + g.writeNumberField("hash", 0); + g.writeEndObject(); + } + baos.write('\n'); + return baos.toByteArray(); + } + + private static byte[] summaryJson(int lastId) throws IOException { + JsonFactory f = new JsonFactory(); + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + try (JsonGenerator g = f.createGenerator(baos)) { + g.writeStartObject(); + g.writeStringField("t", "S"); + g.writeNumberField("lastId", lastId); + g.writeNumberField("crc32", 0L); + g.writeEndObject(); + } + baos.write('\n'); + return baos.toByteArray(); + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderListSegmentsUnreadableTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderListSegmentsUnreadableTest.java new file mode 100644 index 00000000000..d2d1ecec370 --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderListSegmentsUnreadableTest.java @@ -0,0 +1,81 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.file.Files; +import java.nio.file.Path; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import com.fasterxml.jackson.core.JsonFactory; +import com.fasterxml.jackson.core.JsonGenerator; + +/** + * Ensures listSegments tolerates unreadable/mis-typed entries that match the filename pattern by creating a directory + * named like a segment. This exercises the catch(IOException) branch in the segment header read. + */ +class ValueStoreWalReaderListSegmentsUnreadableTest { + + @TempDir + Path tempDir; + + @Test + void unreadableSegmentHeaderIsTolerated() throws Exception { + Path walDir = tempDir.resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + Files.createDirectories(walDir); + // Create a directory that matches the segment filename pattern -> readSegmentSequence will fail to open + Files.createDirectory(walDir.resolve("wal-100.v1")); + // Also create a valid uncompressed segment with sequence 1 so the reader has something to process + Path seg = walDir.resolve("wal-1.v1"); + Files.write(seg, headerFrame(1, 1)); + + ValueStoreWalConfig cfg = ValueStoreWalConfig.builder().walDirectory(walDir).storeUuid("s").build(); + try (ValueStoreWalReader reader = ValueStoreWalReader.open(cfg)) { + ValueStoreWalReader.ScanResult res = reader.scan(); + // Completeness may be false due to a sequence gap introduced by the unreadable item, but no exception + // occurs + assertThat(res.records()).isEmpty(); + } + } + + private static byte[] headerFrame(int seq, int firstId) throws IOException { + JsonFactory f = new JsonFactory(); + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + try (JsonGenerator g = f.createGenerator(baos)) { + g.writeStartObject(); + g.writeStringField("t", "V"); + g.writeNumberField("ver", 1); + g.writeStringField("store", "s"); + g.writeStringField("engine", "valuestore"); + g.writeNumberField("created", 0); + g.writeNumberField("segment", seq); + g.writeNumberField("firstId", firstId); + g.writeEndObject(); + } + baos.write('\n'); + byte[] json = baos.toByteArray(); + ByteBuffer buf = ByteBuffer.allocate(4 + json.length + 4).order(ByteOrder.LITTLE_ENDIAN); + buf.putInt(json.length); + buf.put(json); + buf.putInt(0); // CRC ignored in readSegmentSequence + buf.flip(); + byte[] framed = new byte[buf.remaining()]; + buf.get(framed); + return framed; + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderParseJsonNoStartObjectTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderParseJsonNoStartObjectTest.java new file mode 100644 index 00000000000..aeae06c363f --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderParseJsonNoStartObjectTest.java @@ -0,0 +1,75 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.ByteArrayOutputStream; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.file.Files; +import java.nio.file.Path; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +/** + * Covers the parseJson branch where the first token is not START_OBJECT, by writing a frame with a single newline as + * JSON payload. The reader should ignore the frame and proceed without errors. + */ +class ValueStoreWalReaderParseJsonNoStartObjectTest { + + @TempDir + Path tempDir; + + @Test + void frameNotStartingWithStartObjectIsIgnored() throws Exception { + Path walDir = tempDir.resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + Files.createDirectories(walDir); + Path seg = walDir.resolve("wal-1.v1"); + ByteArrayOutputStream out = new ByteArrayOutputStream(); + // Valid header frame (minimal '{}') with correct CRC + byte[] hdr = new byte[] { '{', '}' }; + out.write(lenLE(hdr.length)); + out.write(hdr); + out.write(intLE(crc32c(hdr))); + // Non-object JSON: just a newline (0x0A) + out.write(lenLE(1)); + out.write(new byte[] { '\n' }); + out.write(intLE(crc32c(new byte[] { '\n' }))); + Files.write(seg, out.toByteArray()); + + ValueStoreWalConfig cfg = ValueStoreWalConfig.builder().walDirectory(walDir).storeUuid("s").build(); + try (ValueStoreWalReader reader = ValueStoreWalReader.open(cfg)) { + ValueStoreWalReader.ScanResult res = reader.scan(); + assertThat(res.records()).isEmpty(); + assertThat(res.complete()).isTrue(); + } + } + + private static byte[] lenLE(int v) { + ByteBuffer b = ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putInt(v); + b.flip(); + byte[] a = new byte[4]; + b.get(a); + return a; + } + + private static int crc32c(byte[] data) { + java.util.zip.CRC32C c = new java.util.zip.CRC32C(); + c.update(data, 0, data.length); + return (int) c.getValue(); + } + + private static byte[] intLE(int v) { + return lenLE(v); + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderParseJsonSkipChildrenTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderParseJsonSkipChildrenTest.java new file mode 100644 index 00000000000..553b3e5bab0 --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderParseJsonSkipChildrenTest.java @@ -0,0 +1,122 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.file.Files; +import java.nio.file.Path; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import com.fasterxml.jackson.core.JsonFactory; +import com.fasterxml.jackson.core.JsonGenerator; + +/** + * Crafts a minted frame with an extra nested object field to exercise parseJson's skipChildren branch. + */ +class ValueStoreWalReaderParseJsonSkipChildrenTest { + + @TempDir + Path tempDir; + + @Test + void mintedWithExtraNestedObjectIsParsedAndIgnored() throws Exception { + Path walDir = tempDir.resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + Files.createDirectories(walDir); + Path seg = walDir.resolve("wal-1.v1"); + ByteArrayOutputStream out = new ByteArrayOutputStream(); + // Header + byte[] hdr = headerJson(1, 1); + out.write(lenLE(hdr.length)); + out.write(hdr); + out.write(intLE(crc32c(hdr))); + // Minted with extra nested object field "x": {"a":1} + byte[] minted = mintedJsonWithExtra(123L, 1); + out.write(lenLE(minted.length)); + out.write(minted); + out.write(intLE(crc32c(minted))); + Files.write(seg, out.toByteArray()); + + ValueStoreWalConfig cfg = ValueStoreWalConfig.builder().walDirectory(walDir).storeUuid("s").build(); + try (ValueStoreWalReader reader = ValueStoreWalReader.open(cfg)) { + ValueStoreWalReader.ScanResult res = reader.scan(); + assertThat(res.complete()).isTrue(); + assertThat(res.records()).hasSize(1); + assertThat(res.records().get(0).id()).isEqualTo(1); + assertThat(res.lastValidLsn()).isEqualTo(123L); + } + } + + private static byte[] headerJson(int segment, int firstId) throws IOException { + JsonFactory f = new JsonFactory(); + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + try (JsonGenerator g = f.createGenerator(baos)) { + g.writeStartObject(); + g.writeStringField("t", "V"); + g.writeNumberField("ver", 1); + g.writeStringField("store", "s"); + g.writeStringField("engine", "valuestore"); + g.writeNumberField("created", 0); + g.writeNumberField("segment", segment); + g.writeNumberField("firstId", firstId); + g.writeEndObject(); + } + baos.write('\n'); + return baos.toByteArray(); + } + + private static byte[] mintedJsonWithExtra(long lsn, int id) throws IOException { + JsonFactory f = new JsonFactory(); + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + try (JsonGenerator g = f.createGenerator(baos)) { + g.writeStartObject(); + g.writeStringField("t", "M"); + g.writeNumberField("lsn", lsn); + g.writeNumberField("id", id); + g.writeStringField("vk", "I"); + g.writeStringField("lex", "http://ex/id" + id); + g.writeStringField("dt", ""); + g.writeStringField("lang", ""); + g.writeNumberField("hash", 0); + // Extra nested object to trigger skipChildren + g.writeObjectFieldStart("x"); + g.writeNumberField("a", 1); + g.writeEndObject(); + g.writeEndObject(); + } + baos.write('\n'); + return baos.toByteArray(); + } + + private static byte[] lenLE(int v) { + ByteBuffer b = ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putInt(v); + b.flip(); + byte[] a = new byte[4]; + b.get(a); + return a; + } + + private static byte[] intLE(int v) { + return lenLE(v); + } + + private static int crc32c(byte[] data) { + java.util.zip.CRC32C c = new java.util.zip.CRC32C(); + c.update(data, 0, data.length); + return (int) c.getValue(); + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderTruncatedRecordTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderTruncatedRecordTest.java new file mode 100644 index 00000000000..06d87bb2b2b --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderTruncatedRecordTest.java @@ -0,0 +1,84 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.file.Files; +import java.nio.file.Path; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import com.fasterxml.jackson.core.JsonFactory; +import com.fasterxml.jackson.core.JsonGenerator; + +/** + * Ensures the reader marks incomplete when a frame is truncated (length OK, payload/CRC missing). + */ +class ValueStoreWalReaderTruncatedRecordTest { + + @TempDir + Path tempDir; + + @Test + void truncatedFrameMarksIncomplete() throws Exception { + Path walDir = tempDir.resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + Files.createDirectories(walDir); + + Path seg = walDir.resolve("wal-1.v1"); + byte[] header = headerFrame(); + // Create a frame header with non-zero length but write no payload/CRC + ByteBuffer len = ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putInt(16); + len.flip(); + ByteArrayOutputStream out = new ByteArrayOutputStream(); + out.write(header); + out.write(len.array(), 0, 4); + Files.write(seg, out.toByteArray()); + + ValueStoreWalConfig cfg = ValueStoreWalConfig.builder().walDirectory(walDir).storeUuid("s").build(); + try (ValueStoreWalReader reader = ValueStoreWalReader.open(cfg)) { + ValueStoreWalReader.ScanResult scan = reader.scan(); + assertThat(scan.complete()).isFalse(); + assertThat(scan.records()).isEmpty(); + } + } + + private static byte[] headerFrame() throws IOException { + JsonFactory f = new JsonFactory(); + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + try (JsonGenerator g = f.createGenerator(baos)) { + g.writeStartObject(); + g.writeStringField("t", "V"); + g.writeNumberField("ver", 1); + g.writeStringField("store", "s"); + g.writeStringField("engine", "valuestore"); + g.writeNumberField("created", 0); + g.writeNumberField("segment", 1); + g.writeNumberField("firstId", 1); + g.writeEndObject(); + } + baos.write('\n'); + byte[] json = baos.toByteArray(); + ByteBuffer frame = ByteBuffer.allocate(4 + json.length + 4).order(ByteOrder.LITTLE_ENDIAN); + frame.putInt(json.length); + frame.put(json); + frame.putInt(0); + frame.flip(); + byte[] framed = new byte[frame.remaining()]; + frame.get(framed); + return framed; + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderUncompressedCrcMismatchTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderUncompressedCrcMismatchTest.java new file mode 100644 index 00000000000..beedd2f07c9 --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderUncompressedCrcMismatchTest.java @@ -0,0 +1,109 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.file.Files; +import java.nio.file.Path; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import com.fasterxml.jackson.core.JsonFactory; +import com.fasterxml.jackson.core.JsonGenerator; + +/** + * Ensures uncompressed reader stops and marks incomplete when CRC32C mismatches. + */ +class ValueStoreWalReaderUncompressedCrcMismatchTest { + + @TempDir + Path tempDir; + + @Test + void crcMismatchStopsScan() throws Exception { + Path walDir = tempDir.resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + Files.createDirectories(walDir); + Path seg = walDir.resolve("wal-1.v1"); + ByteArrayOutputStream out = new ByteArrayOutputStream(); + // Valid header + frame(out, headerJson(1, 1), true); + // Minted record with wrong CRC + frame(out, mintedJson(1L, 1), false); + Files.write(seg, out.toByteArray()); + + ValueStoreWalConfig cfg = ValueStoreWalConfig.builder().walDirectory(walDir).storeUuid("s").build(); + try (ValueStoreWalReader reader = ValueStoreWalReader.open(cfg)) { + ValueStoreWalReader.ScanResult res = reader.scan(); + assertThat(res.complete()).isFalse(); + assertThat(res.records()).isEmpty(); + } + } + + private static void frame(ByteArrayOutputStream out, byte[] json, boolean correctCrc) { + ByteBuffer len = ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putInt(json.length); + len.flip(); + out.write(len.array(), 0, 4); + out.write(json, 0, json.length); + int crc = correctCrc ? crc32c(json) : 0xDEADBEEF; // wrong CRC + ByteBuffer cb = ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putInt(crc); + cb.flip(); + out.write(cb.array(), 0, 4); + } + + private static int crc32c(byte[] data) { + java.util.zip.CRC32C c = new java.util.zip.CRC32C(); + c.update(data, 0, data.length); + return (int) c.getValue(); + } + + private static byte[] headerJson(int segment, int firstId) throws IOException { + JsonFactory f = new JsonFactory(); + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + try (JsonGenerator g = f.createGenerator(baos)) { + g.writeStartObject(); + g.writeStringField("t", "V"); + g.writeNumberField("ver", 1); + g.writeStringField("store", "s"); + g.writeStringField("engine", "valuestore"); + g.writeNumberField("created", 0); + g.writeNumberField("segment", segment); + g.writeNumberField("firstId", firstId); + g.writeEndObject(); + } + baos.write('\n'); + return baos.toByteArray(); + } + + private static byte[] mintedJson(long lsn, int id) throws IOException { + JsonFactory f = new JsonFactory(); + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + try (JsonGenerator g = f.createGenerator(baos)) { + g.writeStartObject(); + g.writeStringField("t", "M"); + g.writeNumberField("lsn", lsn); + g.writeNumberField("id", id); + g.writeStringField("vk", "I"); + g.writeStringField("lex", "http://ex/id" + id); + g.writeStringField("dt", ""); + g.writeStringField("lang", ""); + g.writeNumberField("hash", 0); + g.writeEndObject(); + } + baos.write('\n'); + return baos.toByteArray(); + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderUncompressedTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderUncompressedTest.java new file mode 100644 index 00000000000..3595e73d468 --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderUncompressedTest.java @@ -0,0 +1,134 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.List; +import java.util.UUID; +import java.util.zip.CRC32C; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import com.fasterxml.jackson.core.JsonFactory; +import com.fasterxml.jackson.core.JsonGenerator; + +/** + * Exercises ValueStoreWalReader's uncompressed path by writing a minimal .v1 segment by hand and verifying iteration. + */ +class ValueStoreWalReaderUncompressedTest { + + @TempDir + Path tempDir; + + @Test + void readsMintedRecordsFromUncompressedSegment() throws Exception { + Path walDir = tempDir.resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + Files.createDirectories(walDir); + + // Build a minimal uncompressed segment with header (V) and two minted (M) records + Path seg = walDir.resolve("wal-100.v1"); + byte[] segmentBytes = buildUncompressedSegment("store-" + UUID.randomUUID(), 1, 100); + Files.write(seg, segmentBytes); + + ValueStoreWalConfig cfg = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid("store-irrelevant") + .build(); + + List records = new ArrayList<>(); + try (ValueStoreWalReader reader = ValueStoreWalReader.open(cfg)) { + var it = reader.iterator(); + while (it.hasNext()) { + records.add(it.next()); + } + assertThat(reader.lastValidLsn()).isEqualTo(2L); + assertThat(reader.isComplete()).isTrue(); + } + assertThat(records).hasSize(2); + assertThat(records.get(0).id()).isEqualTo(100); + assertThat(records.get(1).id()).isEqualTo(101); + assertThat(records.get(0).valueKind()).isEqualTo(ValueStoreWalValueKind.IRI); + assertThat(records.get(1).valueKind()).isEqualTo(ValueStoreWalValueKind.LITERAL); + } + + private static byte[] buildUncompressedSegment(String storeUuid, int segmentSeq, int firstId) throws IOException { + ByteArrayOutputStream out = new ByteArrayOutputStream(1024); + // header frame + byte[] hdr = headerJson(storeUuid, segmentSeq, firstId); + frame(out, hdr); + // minted 1 + byte[] m1 = mintedJson(1L, firstId, "I", "http://example.com/x", "", "", 123); + frame(out, m1); + // minted 2 + byte[] m2 = mintedJson(2L, firstId + 1, "L", "hello", "http://www.w3.org/2001/XMLSchema#string", "", 456); + frame(out, m2); + return out.toByteArray(); + } + + private static void frame(ByteArrayOutputStream out, byte[] json) { + ByteBuffer buf = ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putInt(json.length); + buf.flip(); + out.write(buf.array(), 0, 4); + out.write(json, 0, json.length); + CRC32C c = new CRC32C(); + c.update(json, 0, json.length); + ByteBuffer crc = ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putInt((int) c.getValue()); + crc.flip(); + out.write(crc.array(), 0, 4); + } + + private static byte[] headerJson(String store, int segment, int firstId) throws IOException { + JsonFactory f = new JsonFactory(); + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + try (JsonGenerator g = f.createGenerator(baos)) { + g.writeStartObject(); + g.writeStringField("t", "V"); + g.writeNumberField("ver", 1); + g.writeStringField("store", store); + g.writeStringField("engine", "valuestore"); + g.writeNumberField("created", 0); + g.writeNumberField("segment", segment); + g.writeNumberField("firstId", firstId); + g.writeEndObject(); + } + baos.write('\n'); + return baos.toByteArray(); + } + + private static byte[] mintedJson(long lsn, int id, String vk, String lex, String dt, String lang, int hash) + throws IOException { + JsonFactory f = new JsonFactory(); + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + try (JsonGenerator g = f.createGenerator(baos)) { + g.writeStartObject(); + g.writeStringField("t", "M"); + g.writeNumberField("lsn", lsn); + g.writeNumberField("id", id); + g.writeStringField("vk", vk); + g.writeStringField("lex", lex == null ? "" : lex); + g.writeStringField("dt", dt == null ? "" : dt); + g.writeStringField("lang", lang == null ? "" : lang); + g.writeNumberField("hash", hash); + g.writeEndObject(); + } + baos.write('\n'); + return baos.toByteArray(); + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderUnknownValueKindTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderUnknownValueKindTest.java new file mode 100644 index 00000000000..9906c6c4401 --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalReaderUnknownValueKindTest.java @@ -0,0 +1,108 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.junit.jupiter.api.Assertions.assertThrows; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.file.Files; +import java.nio.file.Path; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import com.fasterxml.jackson.core.JsonFactory; +import com.fasterxml.jackson.core.JsonGenerator; + +/** + * Verifies that encountering an unknown value kind code causes parsing to fail with IllegalArgumentException. + */ +class ValueStoreWalReaderUnknownValueKindTest { + + @TempDir + Path tempDir; + + @Test + void unknownValueKindThrows() throws Exception { + Path walDir = tempDir.resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + Files.createDirectories(walDir); + Path seg = walDir.resolve("wal-1.v1"); + ByteArrayOutputStream out = new ByteArrayOutputStream(); + frame(out, headerJson(1, 1)); + frame(out, invalidMintedJson(2L, 2)); + Files.write(seg, out.toByteArray()); + + ValueStoreWalConfig cfg = ValueStoreWalConfig.builder().walDirectory(walDir).storeUuid("s").build(); + try (ValueStoreWalReader reader = ValueStoreWalReader.open(cfg)) { + assertThrows(IllegalArgumentException.class, () -> { + // Trigger parsing by iterating + reader.scan(); + }); + } + } + + private static void frame(ByteArrayOutputStream out, byte[] json) { + ByteBuffer len = ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putInt(json.length); + len.flip(); + out.write(len.array(), 0, 4); + out.write(json, 0, json.length); + int crc = crc32c(json); + ByteBuffer cb = ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putInt(crc); + cb.flip(); + out.write(cb.array(), 0, 4); + } + + private static int crc32c(byte[] data) { + java.util.zip.CRC32C c = new java.util.zip.CRC32C(); + c.update(data, 0, data.length); + return (int) c.getValue(); + } + + private static byte[] headerJson(int segment, int firstId) throws IOException { + JsonFactory f = new JsonFactory(); + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + try (JsonGenerator g = f.createGenerator(baos)) { + g.writeStartObject(); + g.writeStringField("t", "V"); + g.writeNumberField("ver", 1); + g.writeStringField("store", "s"); + g.writeStringField("engine", "valuestore"); + g.writeNumberField("created", 0); + g.writeNumberField("segment", segment); + g.writeNumberField("firstId", firstId); + g.writeEndObject(); + } + baos.write('\n'); + return baos.toByteArray(); + } + + private static byte[] invalidMintedJson(long lsn, int id) throws IOException { + JsonFactory f = new JsonFactory(); + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + try (JsonGenerator g = f.createGenerator(baos)) { + g.writeStartObject(); + g.writeStringField("t", "M"); + g.writeNumberField("lsn", lsn); + g.writeNumberField("id", id); + g.writeStringField("vk", "?"); // invalid code + g.writeStringField("lex", "x"); + g.writeStringField("dt", ""); + g.writeStringField("lang", ""); + g.writeNumberField("hash", 0); + g.writeEndObject(); + } + baos.write('\n'); + return baos.toByteArray(); + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalRecordNormalizationTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalRecordNormalizationTest.java new file mode 100644 index 00000000000..3904e09848d --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalRecordNormalizationTest.java @@ -0,0 +1,26 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; + +import org.junit.jupiter.api.Test; + +class ValueStoreWalRecordNormalizationTest { + + @Test + void nullStringsAreNormalizedToEmpty() { + ValueStoreWalRecord r = new ValueStoreWalRecord(1L, 123, ValueStoreWalValueKind.IRI, null, null, null, 0); + assertThat(r.lexical()).isEqualTo(""); + assertThat(r.datatype()).isEqualTo(""); + assertThat(r.language()).isEqualTo(""); + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalRecoveryCorruptionTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalRecoveryCorruptionTest.java new file mode 100644 index 00000000000..946f3c05049 --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalRecoveryCorruptionTest.java @@ -0,0 +1,288 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ + +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.BufferedOutputStream; +import java.io.DataOutputStream; +import java.io.File; +import java.io.FileOutputStream; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.StandardOpenOption; +import java.util.LinkedHashMap; +import java.util.Map; +import java.util.Set; +import java.util.UUID; + +import org.eclipse.rdf4j.common.io.ByteArrayUtil; +import org.eclipse.rdf4j.model.IRI; +import org.eclipse.rdf4j.model.Literal; +import org.eclipse.rdf4j.model.ValueFactory; +import org.eclipse.rdf4j.model.impl.SimpleValueFactory; +import org.eclipse.rdf4j.model.util.Values; +import org.eclipse.rdf4j.model.vocabulary.RDF; +import org.eclipse.rdf4j.model.vocabulary.RDFS; +import org.eclipse.rdf4j.repository.Repository; +import org.eclipse.rdf4j.repository.RepositoryConnection; +import org.eclipse.rdf4j.repository.sail.SailRepository; +import org.eclipse.rdf4j.sail.nativerdf.NativeStore; +import org.eclipse.rdf4j.sail.nativerdf.ValueStore; +import org.eclipse.rdf4j.sail.nativerdf.datastore.DataStore; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.Timeout; +import org.junit.jupiter.api.io.TempDir; + +/** + * Tests that corrupt or missing ValueStore files can be reconstructed from the ValueStore WAL, restoring consistent IDs + * so existing triple indexes remain valid. + */ +class ValueStoreWalRecoveryCorruptionTest { + + private static final ValueFactory VF = SimpleValueFactory.getInstance(); + + @TempDir + Path tempDir; + + @Test + @Timeout(10) + void rebuildsAfterDeletingAllValueFiles() throws Exception { + File dataDir = tempDir.resolve("store").toFile(); + dataDir.mkdirs(); + + // Pre-create an empty context index to avoid ContextStore reconstruction during init + ensureEmptyContextIndex(dataDir.toPath()); + Repository repo = new SailRepository(new NativeStore(dataDir, "spoc,posc")); + repo.init(); + + try (RepositoryConnection conn = repo.getConnection()) { + conn.begin(); + IRI exA = VF.createIRI("http://example.org/a0"); + IRI exB = VF.createIRI("http://example.org/b1"); + IRI exC = VF.createIRI("http://example.org/c2"); + Literal lit0 = VF.createLiteral("zero"); + Literal lit1 = VF.createLiteral("one"); + Literal lit2 = VF.createLiteral("two"); + Literal lit2en = VF.createLiteral("two", "en"); + Literal litTyped = VF.createLiteral(1.2); + + conn.add(exA, RDFS.LABEL, lit0); + conn.add(exB, RDFS.LABEL, lit1, VF.createIRI("urn:one")); + conn.add(exC, RDFS.LABEL, lit2, VF.createIRI("urn:two")); + conn.add(exC, RDFS.LABEL, lit2, VF.createIRI("urn:two")); + conn.add(Values.bnode(), RDF.TYPE, Values.bnode(), VF.createIRI("urn:two")); + conn.add(exC, RDFS.LABEL, lit2en, VF.createIRI("urn:two")); + conn.add(exC, RDFS.LABEL, litTyped, VF.createIRI("urn:two")); + conn.commit(); + } + + repo.shutDown(); + + // Simulate corruption: delete all ValueStore files + deleteIfExists(dataDir.toPath().resolve("values.dat")); + deleteIfExists(dataDir.toPath().resolve("values.id")); + deleteIfExists(dataDir.toPath().resolve("values.hash")); + + recoverValueStoreFromWal(dataDir.toPath()); + validateDictionaryMatchesWal(dataDir.toPath()); + } + + @Test + @Timeout(10) + void rebuildsAfterCorruptingValuesDat() throws Exception { + File dataDir = tempDir.resolve("store2").toFile(); + dataDir.mkdirs(); + + ensureEmptyContextIndex(dataDir.toPath()); + Repository repo = new SailRepository(new NativeStore(dataDir, "spoc,posc")); + repo.init(); + try (RepositoryConnection conn = repo.getConnection()) { + conn.begin(); + conn.add(VF.createIRI("http://ex.com/s"), RDFS.LABEL, VF.createLiteral("hello")); + conn.add(VF.createIRI("http://ex.com/t"), RDFS.LABEL, VF.createLiteral("world", "en")); + conn.add(VF.createIRI("http://ex.com/u"), RDFS.LABEL, VF.createLiteral(42)); + conn.commit(); + } + repo.shutDown(); + + Path valuesDat = dataDir.toPath().resolve("values.dat"); + if (Files.exists(valuesDat)) { + Files.newByteChannel(valuesDat, Set.of(StandardOpenOption.WRITE)) + .truncate(0) + .close(); + } + + recoverValueStoreFromWal(dataDir.toPath()); + validateDictionaryMatchesWal(dataDir.toPath()); + } + + private void deleteIfExists(Path path) throws IOException { + if (Files.exists(path)) { + Files.delete(path); + } + } + + private void recoverValueStoreFromWal(Path dataDir) throws Exception { + Path walDir = dataDir.resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + Path uuidFile = walDir.resolve("store.uuid"); + String storeUuid = Files.exists(uuidFile) ? Files.readString(uuidFile, StandardCharsets.UTF_8).trim() + : UUID.randomUUID().toString(); + + ValueStoreWalConfig config = ValueStoreWalConfig.builder().walDirectory(walDir).storeUuid(storeUuid).build(); + + Map dictionary; + try (ValueStoreWalReader reader = ValueStoreWalReader.open(config)) { + ValueStoreWalRecovery recovery = new ValueStoreWalRecovery(); + dictionary = new LinkedHashMap<>(recovery.replay(reader)); + } + + try (DataStore ds = new DataStore(dataDir.toFile(), "values", false)) { + for (ValueStoreWalRecord record : dictionary.values()) { + switch (record.valueKind()) { + case NAMESPACE: { + byte[] nsBytes = record.lexical().getBytes(StandardCharsets.UTF_8); + ds.storeData(nsBytes); + break; + } + case IRI: { + byte[] iriBytes = encodeIri(record.lexical(), ds); + ds.storeData(iriBytes); + break; + } + case BNODE: { + byte[] idData = record.lexical().getBytes(StandardCharsets.UTF_8); + byte[] bnode = new byte[1 + idData.length]; + bnode[0] = 0x2; + ByteArrayUtil.put(idData, bnode, 1); + ds.storeData(bnode); + break; + } + case LITERAL: { + byte[] litBytes = encodeLiteral(record.lexical(), record.datatype(), record.language(), ds); + ds.storeData(litBytes); + break; + } + default: + break; + } + } + ds.sync(); + } + } + + private byte[] encodeIri(String lexical, DataStore ds) throws IOException { + IRI iri = VF.createIRI(lexical); + String ns = iri.getNamespace(); + String local = iri.getLocalName(); + int nsId = ds.getID(ns.getBytes(StandardCharsets.UTF_8)); + if (nsId == -1) { + nsId = ds.storeData(ns.getBytes(StandardCharsets.UTF_8)); + } + byte[] localBytes = local.getBytes(StandardCharsets.UTF_8); + byte[] data = new byte[1 + 4 + localBytes.length]; + data[0] = 0x1; + ByteArrayUtil.putInt(nsId, data, 1); + ByteArrayUtil.put(localBytes, data, 5); + return data; + } + + private byte[] encodeLiteral(String label, String datatype, String language, DataStore ds) throws IOException { + int dtId = -1; // -1 denotes UNKNOWN_ID + if (datatype != null && !datatype.isEmpty()) { + byte[] dtBytes = encodeIri(datatype, ds); + int id = ds.getID(dtBytes); + dtId = id == -1 ? ds.storeData(dtBytes) : id; + } + byte[] langBytes = language == null ? new byte[0] : language.getBytes(StandardCharsets.UTF_8); + byte[] labelBytes = label.getBytes(StandardCharsets.UTF_8); + byte[] data = new byte[1 + 4 + 1 + langBytes.length + labelBytes.length]; + data[0] = 0x3; + ByteArrayUtil.putInt(dtId, data, 1); + data[5] = (byte) (langBytes.length & 0xFF); + if (langBytes.length > 0) { + ByteArrayUtil.put(langBytes, data, 6); + } + ByteArrayUtil.put(labelBytes, data, 6 + langBytes.length); + return data; + } + + private void validateDictionaryMatchesWal(Path dataDir) throws Exception { + Path walDir = dataDir.resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + String storeUuid = Files.readString(walDir.resolve("store.uuid"), StandardCharsets.UTF_8).trim(); + ValueStoreWalConfig config = ValueStoreWalConfig.builder().walDirectory(walDir).storeUuid(storeUuid).build(); + + Map dictionary; + try (ValueStoreWalReader reader = ValueStoreWalReader.open(config)) { + ValueStoreWalRecovery recovery = new ValueStoreWalRecovery(); + dictionary = new LinkedHashMap<>(recovery.replay(reader)); + } + + try (ValueStore vs = new ValueStore(dataDir.toFile(), false, ValueStore.VALUE_CACHE_SIZE, + ValueStore.VALUE_ID_CACHE_SIZE, ValueStore.NAMESPACE_CACHE_SIZE, ValueStore.NAMESPACE_ID_CACHE_SIZE, + null)) { + for (ValueStoreWalRecord record : dictionary.values()) { + switch (record.valueKind()) { + case IRI: { + IRI iri = VF.createIRI(record.lexical()); + int id = vs.getID(iri); + assertThat(id).isNotEqualTo(-1); + assertThat(vs.getValue(id).stringValue()).isEqualTo(record.lexical()); + break; + } + case BNODE: { + int id = vs.getID(VF.createBNode(record.lexical())); + assertThat(id).isNotEqualTo(-1); + assertThat(vs.getValue(id).stringValue()).isEqualTo(record.lexical()); + break; + } + case LITERAL: { + Literal lit; + if (record.language() != null && !record.language().isEmpty()) { + lit = VF.createLiteral(record.lexical(), record.language()); + } else if (record.datatype() != null && !record.datatype().isEmpty()) { + lit = VF.createLiteral(record.lexical(), VF.createIRI(record.datatype())); + } else { + lit = VF.createLiteral(record.lexical()); + } + int id = vs.getID(lit); + assertThat(id).isNotEqualTo(-1); + assertThat(vs.getValue(id).stringValue()).isEqualTo(lit.stringValue()); + break; + } + case NAMESPACE: + // Namespaces indirectly validated via IRIs + break; + default: + break; + } + } + } + } + + private void ensureEmptyContextIndex(Path dataDir) throws IOException { + Path file = dataDir.resolve("contexts.dat"); + if (Files.exists(file)) { + return; + } + Files.createDirectories(dataDir); + try (var out = new DataOutputStream( + new BufferedOutputStream(new FileOutputStream(file.toFile())))) { + out.write(new byte[] { 'n', 'c', 'f' }); + out.writeByte(1); + out.writeInt(0); + out.flush(); + } + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalRecoveryDedupTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalRecoveryDedupTest.java new file mode 100644 index 00000000000..388275af3ba --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalRecoveryDedupTest.java @@ -0,0 +1,116 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Map; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import com.fasterxml.jackson.core.JsonFactory; +import com.fasterxml.jackson.core.JsonGenerator; + +/** + * Ensures ValueStoreWalRecovery keeps the first occurrence of a duplicated id encountered across segments. + */ +class ValueStoreWalRecoveryDedupTest { + + @TempDir + Path tempDir; + + @Test + void keepsFirstOccurrenceOfId() throws Exception { + Path walDir = tempDir.resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + Files.createDirectories(walDir); + + // Segment seq=1 with id=100 lex="first" + Files.write(walDir.resolve("wal-100.v1"), segmentBytes(1, 100, "first")); + // Segment seq=2 with id=100 lex="second" + Files.write(walDir.resolve("wal-200.v1"), segmentBytes(2, 100, "second")); + + ValueStoreWalConfig cfg = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid("s") + .build(); + Map dict; + try (ValueStoreWalReader reader = ValueStoreWalReader.open(cfg)) { + dict = new ValueStoreWalRecovery().replay(reader); + } + assertThat(dict).containsKey(100); + assertThat(dict.get(100).lexical()).isEqualTo("first"); // first occurrence retained + } + + private static byte[] segmentBytes(int segment, int id, String lex) throws IOException { + ByteArrayOutputStream out = new ByteArrayOutputStream(); + frame(out, header(segment, id)); + frame(out, minted(1L, id, lex)); + return out.toByteArray(); + } + + private static byte[] header(int segment, int firstId) throws IOException { + JsonFactory f = new JsonFactory(); + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + try (JsonGenerator g = f.createGenerator(baos)) { + g.writeStartObject(); + g.writeStringField("t", "V"); + g.writeNumberField("ver", 1); + g.writeStringField("store", "s"); + g.writeStringField("engine", "valuestore"); + g.writeNumberField("created", 0); + g.writeNumberField("segment", segment); + g.writeNumberField("firstId", firstId); + g.writeEndObject(); + } + baos.write('\n'); + return baos.toByteArray(); + } + + private static byte[] minted(long lsn, int id, String lex) throws IOException { + JsonFactory f = new JsonFactory(); + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + try (JsonGenerator g = f.createGenerator(baos)) { + g.writeStartObject(); + g.writeStringField("t", "M"); + g.writeNumberField("lsn", lsn); + g.writeNumberField("id", id); + g.writeStringField("vk", "I"); + g.writeStringField("lex", lex); + g.writeStringField("dt", ""); + g.writeStringField("lang", ""); + g.writeNumberField("hash", 0); + g.writeEndObject(); + } + baos.write('\n'); + return baos.toByteArray(); + } + + private static void frame(ByteArrayOutputStream out, byte[] json) { + ByteBuffer len = ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putInt(json.length); + len.flip(); + out.write(len.array(), 0, 4); + out.write(json, 0, json.length); + int crc = java.util.zip.CRC32C.class.desiredAssertionStatus() ? 0 : 0; // keep import minimal + java.util.zip.CRC32C c = new java.util.zip.CRC32C(); + c.update(json, 0, json.length); + crc = (int) c.getValue(); + ByteBuffer crcBuf = ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putInt(crc); + crcBuf.flip(); + out.write(crcBuf.array(), 0, 4); + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalRecoveryRebuildTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalRecoveryRebuildTest.java new file mode 100644 index 00000000000..9efc99a1e68 --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalRecoveryRebuildTest.java @@ -0,0 +1,205 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ + +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.File; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.OptionalLong; +import java.util.UUID; +import java.util.stream.Collectors; + +import org.eclipse.rdf4j.common.io.ByteArrayUtil; +import org.eclipse.rdf4j.model.IRI; +import org.eclipse.rdf4j.model.Literal; +import org.eclipse.rdf4j.model.ValueFactory; +import org.eclipse.rdf4j.model.impl.SimpleValueFactory; +import org.eclipse.rdf4j.sail.nativerdf.ValueStore; +import org.eclipse.rdf4j.sail.nativerdf.datastore.DataStore; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +class ValueStoreWalRecoveryRebuildTest { + + private static final ValueFactory VF = SimpleValueFactory.getInstance(); + + @TempDir + Path tempDir; + + @Test + void rebuildAssignsExactIds() throws Exception { + Path walDir = tempDir.resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + Files.createDirectories(walDir); + ValueStoreWalConfig config = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid(UUID.randomUUID().toString()) + .build(); + + IRI iri = VF.createIRI("http://example.com/res"); + Literal lit = VF.createLiteral("value", "en"); + + // Mint values and persist WAL + try (ValueStoreWAL wal = ValueStoreWAL.open(config)) { + File valueDir = tempDir.resolve("values").toFile(); + Files.createDirectories(valueDir.toPath()); + try (ValueStore store = new ValueStore(valueDir, false, ValueStore.VALUE_CACHE_SIZE, + ValueStore.VALUE_ID_CACHE_SIZE, ValueStore.NAMESPACE_CACHE_SIZE, + ValueStore.NAMESPACE_ID_CACHE_SIZE, wal)) { + store.storeValue(iri); + store.storeValue(lit); + var lsn = store.drainPendingWalHighWaterMark(); + assertThat(lsn).isPresent(); + wal.awaitDurable(lsn.getAsLong()); + } + } + + Map dictionary; + try (ValueStoreWalReader reader = ValueStoreWalReader.open(config)) { + ValueStoreWalRecovery recovery = new ValueStoreWalRecovery(); + dictionary = new LinkedHashMap<>(recovery.replay(reader)); + } + assertThat(dictionary).isNotEmpty(); + + // Rebuild DataStore directly from WAL dictionary + File dataDir = tempDir.resolve("rebuilt").toFile(); + Files.createDirectories(dataDir.toPath()); + try (DataStore ds = new DataStore(dataDir, "values", false)) { + for (ValueStoreWalRecord rec : dictionary.values()) { + if (rec.valueKind() == ValueStoreWalValueKind.NAMESPACE) { + ds.storeData(rec.lexical().getBytes(StandardCharsets.UTF_8)); + } else if (rec.valueKind() == ValueStoreWalValueKind.IRI) { + ds.storeData(encodeIri(rec.lexical(), ds)); + } else if (rec.valueKind() == ValueStoreWalValueKind.BNODE) { + byte[] idData = rec.lexical().getBytes(StandardCharsets.UTF_8); + byte[] bnode = new byte[1 + idData.length]; + bnode[0] = 0x2; // BNODE tag + ByteArrayUtil.put(idData, bnode, 1); + ds.storeData(bnode); + } else if (rec.valueKind() == ValueStoreWalValueKind.LITERAL) { + ds.storeData(encodeLiteral(rec.lexical(), rec.datatype(), rec.language(), ds)); + } + } + ds.sync(); + } + + // Verify exact id equality using ValueStore on rebuilt data + try (ValueStore vs = new ValueStore(dataDir, false, ValueStore.VALUE_CACHE_SIZE, + ValueStore.VALUE_ID_CACHE_SIZE, ValueStore.NAMESPACE_CACHE_SIZE, + ValueStore.NAMESPACE_ID_CACHE_SIZE, null)) { + for (ValueStoreWalRecord rec : dictionary.values()) { + switch (rec.valueKind()) { + case IRI: + assertThat(vs.getID(VF.createIRI(rec.lexical()))).isEqualTo(rec.id()); + break; + case BNODE: + assertThat(vs.getID(VF.createBNode(rec.lexical()))).isEqualTo(rec.id()); + break; + case LITERAL: + Literal l = (rec.language() != null && !rec.language().isEmpty()) + ? VF.createLiteral(rec.lexical(), rec.language()) + : (rec.datatype() != null && !rec.datatype().isEmpty()) + ? VF.createLiteral(rec.lexical(), VF.createIRI(rec.datatype())) + : VF.createLiteral(rec.lexical()); + assertThat(vs.getID(l)).isEqualTo(rec.id()); + break; + default: + // skip NAMESPACE here + } + } + } + } + + @Test + void missingSegmentMarksIncomplete() throws Exception { + Path walDir = tempDir.resolve("wal-missing"); + Files.createDirectories(walDir); + ValueStoreWalConfig config = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid(UUID.randomUUID().toString()) + .maxSegmentBytes(1 << 12) + .build(); + + Path valueDir = tempDir.resolve("values-missing"); + Files.createDirectories(valueDir); + try (ValueStoreWAL wal = ValueStoreWAL.open(config); + ValueStore store = new ValueStore(valueDir.toFile(), false, ValueStore.VALUE_CACHE_SIZE, + ValueStore.VALUE_ID_CACHE_SIZE, ValueStore.NAMESPACE_CACHE_SIZE, + ValueStore.NAMESPACE_ID_CACHE_SIZE, wal)) { + for (int i = 0; i < 200; i++) { + store.storeValue(VF.createIRI("http://example.com/value/" + i)); + } + OptionalLong lsn = store.drainPendingWalHighWaterMark(); + if (lsn.isPresent()) { + store.awaitWalDurable(lsn.getAsLong()); + } + } + + List segments; + try (var stream = Files.list(walDir)) { + segments = stream.filter(p -> p.getFileName().toString().startsWith("wal-")) + .sorted() + .collect(Collectors.toList()); + } + assertThat(segments).hasSizeGreaterThan(1); + Files.deleteIfExists(segments.get(0)); + + ValueStoreWalRecovery recovery = new ValueStoreWalRecovery(); + ValueStoreWalRecovery.ReplayReport report; + try (ValueStoreWalReader reader = ValueStoreWalReader.open(config)) { + report = recovery.replayWithReport(reader); + } + assertThat(report.complete()).isFalse(); + } + + private byte[] encodeIri(String lexical, DataStore ds) throws IOException { + IRI iri = VF.createIRI(lexical); + String ns = iri.getNamespace(); + String local = iri.getLocalName(); + int nsId = ds.getID(ns.getBytes(StandardCharsets.UTF_8)); + if (nsId == -1) { + nsId = ds.storeData(ns.getBytes(StandardCharsets.UTF_8)); + } + byte[] localBytes = local.getBytes(StandardCharsets.UTF_8); + byte[] data = new byte[1 + 4 + localBytes.length]; + data[0] = 0x1; // URI tag + ByteArrayUtil.putInt(nsId, data, 1); + ByteArrayUtil.put(localBytes, data, 5); + return data; + } + + private byte[] encodeLiteral(String label, String datatype, String language, DataStore ds) throws IOException { + int dtId = -1; // UNKNOWN_ID + if (datatype != null && !datatype.isEmpty()) { + byte[] dtBytes = encodeIri(datatype, ds); + int id = ds.getID(dtBytes); + dtId = id == -1 ? ds.storeData(dtBytes) : id; + } + byte[] langBytes = language == null ? new byte[0] : language.getBytes(StandardCharsets.UTF_8); + byte[] labelBytes = label.getBytes(StandardCharsets.UTF_8); + byte[] data = new byte[1 + 4 + 1 + langBytes.length + labelBytes.length]; + data[0] = 0x3; // LITERAL tag + ByteArrayUtil.putInt(dtId, data, 1); + data[5] = (byte) (langBytes.length & 0xFF); + if (langBytes.length > 0) { + ByteArrayUtil.put(langBytes, data, 6); + } + ByteArrayUtil.put(labelBytes, data, 6 + langBytes.length); + return data; + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalSearchEdgeCasesTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalSearchEdgeCasesTest.java new file mode 100644 index 00000000000..2f337b07a6e --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalSearchEdgeCasesTest.java @@ -0,0 +1,138 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.zip.CRC32C; +import java.util.zip.GZIPOutputStream; + +import org.eclipse.rdf4j.model.Value; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import com.fasterxml.jackson.core.JsonFactory; +import com.fasterxml.jackson.core.JsonGenerator; + +class ValueStoreWalSearchEdgeCasesTest { + + @TempDir + Path tempDir; + + @Test + void returnsNullWhenIdOutsideRange() throws Exception { + Path walDir = tempDir.resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + Files.createDirectories(walDir); + // Segment with firstId=10 and minted ids 10, 20 + Files.write(walDir.resolve("wal-10.v1"), segmentWithTwoIds(1, 10, 10, 20)); + + ValueStoreWalConfig cfg = ValueStoreWalConfig.builder().walDirectory(walDir).storeUuid("s").build(); + ValueStoreWalSearch search = ValueStoreWalSearch.open(cfg); + Value vLow = search.findValueById(5); // before first + Value vHigh = search.findValueById(100); // after last + assertThat(vLow).isNull(); + assertThat(vHigh).isNull(); + } + + @Test + void refreshesSegmentCacheAfterRotation() throws Exception { + Path walDir = tempDir.resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + Files.createDirectories(walDir); + byte[] segment = segmentWithTwoIds(1, 10, 10, 20); + Path plainSegment = walDir.resolve("wal-10.v1"); + Files.write(plainSegment, segment); + + ValueStoreWalConfig cfg = ValueStoreWalConfig.builder().walDirectory(walDir).storeUuid("s").build(); + ValueStoreWalSearch search = ValueStoreWalSearch.open(cfg); + + Value initial = search.findValueById(20); + assertThat(initial).isNotNull(); + + Path gzSegment = walDir.resolve("wal-10.v1.gz"); + Files.write(gzSegment, gzip(segment)); + Files.deleteIfExists(plainSegment); + + Value rotated = search.findValueById(20); + assertThat(rotated).isNotNull(); + assertThat(rotated).isEqualTo(initial); + } + + private static byte[] segmentWithTwoIds(int seq, int firstId, int id1, int id2) throws IOException { + ByteArrayOutputStream out = new ByteArrayOutputStream(); + frame(out, header(seq, firstId)); + frame(out, minted(id1, "I", "http://ex/i" + id1)); + frame(out, minted(id2, "I", "http://ex/i" + id2)); + return out.toByteArray(); + } + + private static byte[] header(int seq, int firstId) throws IOException { + JsonFactory f = new JsonFactory(); + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + try (JsonGenerator g = f.createGenerator(baos)) { + g.writeStartObject(); + g.writeStringField("t", "V"); + g.writeNumberField("ver", 1); + g.writeStringField("store", "s"); + g.writeStringField("engine", "valuestore"); + g.writeNumberField("created", 0); + g.writeNumberField("segment", seq); + g.writeNumberField("firstId", firstId); + g.writeEndObject(); + } + baos.write('\n'); + return baos.toByteArray(); + } + + private static byte[] minted(int id, String vk, String lex) throws IOException { + JsonFactory f = new JsonFactory(); + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + try (JsonGenerator g = f.createGenerator(baos)) { + g.writeStartObject(); + g.writeStringField("t", "M"); + g.writeNumberField("lsn", id); // monotonic for simplicity + g.writeNumberField("id", id); + g.writeStringField("vk", vk); + g.writeStringField("lex", lex); + g.writeStringField("dt", ""); + g.writeStringField("lang", ""); + g.writeNumberField("hash", 0); + g.writeEndObject(); + } + baos.write('\n'); + return baos.toByteArray(); + } + + private static void frame(ByteArrayOutputStream out, byte[] json) { + ByteBuffer length = ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putInt(json.length); + length.flip(); + out.write(length.array(), 0, 4); + out.write(json, 0, json.length); + CRC32C crc32c = new CRC32C(); + crc32c.update(json, 0, json.length); + ByteBuffer crc = ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putInt((int) crc32c.getValue()); + crc.flip(); + out.write(crc.array(), 0, 4); + } + + private static byte[] gzip(byte[] data) throws IOException { + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + try (GZIPOutputStream gzip = new GZIPOutputStream(baos)) { + gzip.write(data); + } + return baos.toByteArray(); + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalSearchTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalSearchTest.java new file mode 100644 index 00000000000..7597f20efc6 --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalSearchTest.java @@ -0,0 +1,80 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ + +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.File; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Map; +import java.util.Random; + +import org.eclipse.rdf4j.model.Value; +import org.eclipse.rdf4j.repository.sail.SailRepository; +import org.eclipse.rdf4j.repository.sail.SailRepositoryConnection; +import org.eclipse.rdf4j.rio.RDFFormat; +import org.eclipse.rdf4j.sail.nativerdf.NativeStore; +import org.eclipse.rdf4j.sail.nativerdf.ValueStore; +import org.eclipse.rdf4j.sail.nativerdf.config.NativeStoreConfig; +import org.eclipse.rdf4j.sail.nativerdf.config.NativeStoreFactory; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +class ValueStoreWalSearchTest { + + @TempDir + File dataDir; + + @Test + void findsValueByIdViaSegmentProbe() throws Exception { + // Configure NativeStore with small WAL segment size to ensure multiple segments possible + NativeStoreConfig cfg = new NativeStoreConfig("spoc,ospc,psoc"); + cfg.setWalMaxSegmentBytes(64 * 1024); // 64 KiB + NativeStore store = (NativeStore) new NativeStoreFactory().getSail(cfg); + store.setDataDir(dataDir); + SailRepository repo = new SailRepository(store); + repo.init(); + try (SailRepositoryConnection conn = repo.getConnection()) { + try (var in = getClass().getClassLoader().getResourceAsStream("benchmarkFiles/datagovbe-valid.ttl")) { + assertThat(in).isNotNull(); + conn.add(in, "", RDFFormat.TURTLE); + } + } + repo.shutDown(); + + Path walDir = dataDir.toPath().resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + String storeUuid = Files.readString(walDir.resolve("store.uuid"), StandardCharsets.UTF_8).trim(); + ValueStoreWalConfig cfgRead = ValueStoreWalConfig.builder().walDirectory(walDir).storeUuid(storeUuid).build(); + + // Build dictionary of minted values from WAL and pick a random entry + Map dict; + try (ValueStoreWalReader reader = ValueStoreWalReader.open(cfgRead)) { + dict = new ValueStoreWalRecovery().replay(reader); + } + assertThat(dict).isNotEmpty(); + Integer[] ids = dict.keySet().toArray(Integer[]::new); + Integer pickId = ids[new Random().nextInt(ids.length)]; + + ValueStoreWalSearch search = ValueStoreWalSearch.open(cfgRead); + Value found = search.findValueById(pickId); + assertThat(found).as("ValueStoreWalSearch should find value by id").isNotNull(); + + // Cross-check against ValueStore + try (ValueStore vs = new ValueStore(dataDir, false)) { + Value vsValue = vs.getValue(pickId); + assertThat(vsValue).isNotNull(); + assertThat(found).isEqualTo(vsValue); + } + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalTestUtils.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalTestUtils.java new file mode 100644 index 00000000000..c7e8bc1a2f8 --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalTestUtils.java @@ -0,0 +1,80 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ + +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import java.io.ByteArrayInputStream; +import java.io.IOException; +import java.io.InputStream; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.zip.GZIPInputStream; + +import com.fasterxml.jackson.core.JsonFactory; +import com.fasterxml.jackson.core.JsonParser; +import com.fasterxml.jackson.core.JsonToken; + +/** + * Test utility helpers for inspecting ValueStore WAL segments. + */ +public final class ValueStoreWalTestUtils { + + private static final JsonFactory JSON_FACTORY = new JsonFactory(); + + private ValueStoreWalTestUtils() { + } + + public static int readSegmentSequence(Path segmentPath) throws IOException { + boolean compressed = segmentPath.getFileName().toString().endsWith(".gz"); + try (InputStream raw = Files.newInputStream(segmentPath); + InputStream in = compressed ? new GZIPInputStream(raw) : raw) { + return readSegmentSequence(in); + } + } + + public static int readSegmentSequence(byte[] segmentContent) throws IOException { + try (ByteArrayInputStream in = new ByteArrayInputStream(segmentContent)) { + return readSegmentSequence(in); + } + } + + private static int readSegmentSequence(InputStream in) throws IOException { + byte[] lenBytes = in.readNBytes(Integer.BYTES); + if (lenBytes.length < Integer.BYTES) { + return 0; + } + ByteBuffer lenBuf = ByteBuffer.wrap(lenBytes).order(ByteOrder.LITTLE_ENDIAN); + int frameLen = lenBuf.getInt(); + if (frameLen <= 0) { + return 0; + } + byte[] jsonBytes = in.readNBytes(frameLen); + if (jsonBytes.length < frameLen) { + return 0; + } + // Skip header CRC + in.readNBytes(Integer.BYTES); + try (JsonParser parser = JSON_FACTORY.createParser(jsonBytes)) { + while (parser.nextToken() != JsonToken.END_OBJECT) { + if (parser.currentToken() == JsonToken.FIELD_NAME) { + String field = parser.getCurrentName(); + parser.nextToken(); + if ("segment".equals(field)) { + return parser.getIntValue(); + } + } + } + } + return 0; + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalThroughputBenchmark.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalThroughputBenchmark.java new file mode 100644 index 00000000000..30a35cfca78 --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalThroughputBenchmark.java @@ -0,0 +1,111 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ + +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.UUID; +import java.util.concurrent.ThreadLocalRandom; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; + +import org.openjdk.jmh.annotations.Benchmark; +import org.openjdk.jmh.annotations.BenchmarkMode; +import org.openjdk.jmh.annotations.Fork; +import org.openjdk.jmh.annotations.Level; +import org.openjdk.jmh.annotations.Measurement; +import org.openjdk.jmh.annotations.Mode; +import org.openjdk.jmh.annotations.OutputTimeUnit; +import org.openjdk.jmh.annotations.Param; +import org.openjdk.jmh.annotations.Scope; +import org.openjdk.jmh.annotations.Setup; +import org.openjdk.jmh.annotations.State; +import org.openjdk.jmh.annotations.TearDown; +import org.openjdk.jmh.annotations.Threads; + +@BenchmarkMode(Mode.Throughput) +@OutputTimeUnit(TimeUnit.SECONDS) +@Measurement(iterations = 3) +@Fork(1) +@State(Scope.Benchmark) +public class ValueStoreWalThroughputBenchmark { + + @Param({ "COMMIT", "INTERVAL", "ALWAYS" }) + public String syncPolicy; + + @Param({ "32", "256" }) + public int payloadBytes; + + @Param({ "0", "1000" }) + public int ackEvery; + + private ValueStoreWalConfig config; + private ValueStoreWAL wal; + private String lexical; + private final AtomicInteger seq = new AtomicInteger(); + + @Setup(Level.Trial) + public void setup() throws IOException { + Path walDir = Files.createTempDirectory("wal-bench-"); + ValueStoreWalConfig.Builder builder = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid(UUID.randomUUID().toString()); + builder.syncPolicy(ValueStoreWalConfig.SyncPolicy.valueOf(syncPolicy)); + config = builder.build(); + wal = ValueStoreWAL.open(config); + lexical = randomAscii(payloadBytes); + } + + @TearDown(Level.Trial) + public void tearDown() throws IOException { + if (wal != null) { + wal.close(); + } + } + + @Benchmark + @Threads(8) + public void logMint_literal() throws IOException, InterruptedException { + int id = seq.incrementAndGet(); + long lsn = wal.logMint(id, ValueStoreWalValueKind.LITERAL, lexical, "", "", 0); + if (ackEvery > 0) { + // acknowledge durability occasionally + if ((id % ackEvery) == 0) { + wal.awaitDurable(lsn); + } + } + } + + @Benchmark + @Threads(8) + public void logMint_iri() throws IOException, InterruptedException { + int id = seq.incrementAndGet(); + long lsn = wal.logMint(id, ValueStoreWalValueKind.IRI, "http://example.com/" + id, "", "", 0); + if (ackEvery > 0) { + if ((id % ackEvery) == 0) { + wal.awaitDurable(lsn); + } + } + } + + private static String randomAscii(int len) { + StringBuilder sb = new StringBuilder(len); + ThreadLocalRandom r = ThreadLocalRandom.current(); + for (int i = 0; i < len; i++) { + // printable ASCII range 32..126 + char c = (char) r.nextInt(32, 127); + sb.append(c); + } + return sb.toString(); + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalValueKindTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalValueKindTest.java new file mode 100644 index 00000000000..a6d08aba63d --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/ValueStoreWalValueKindTest.java @@ -0,0 +1,40 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +import org.junit.jupiter.api.Test; + +class ValueStoreWalValueKindTest { + + @Test + void mapsKnownCodes() { + assertThat(ValueStoreWalValueKind.fromCode("I")).isEqualTo(ValueStoreWalValueKind.IRI); + assertThat(ValueStoreWalValueKind.fromCode("B")).isEqualTo(ValueStoreWalValueKind.BNODE); + assertThat(ValueStoreWalValueKind.fromCode("L")).isEqualTo(ValueStoreWalValueKind.LITERAL); + assertThat(ValueStoreWalValueKind.fromCode("N")).isEqualTo(ValueStoreWalValueKind.NAMESPACE); + } + + @Test + void rejectsUnknownOrEmptyCodes() { + assertThatThrownBy(() -> ValueStoreWalValueKind.fromCode("?")) + .isInstanceOf(IllegalArgumentException.class) + .hasMessageContaining("Unknown value kind code"); + assertThatThrownBy(() -> ValueStoreWalValueKind.fromCode("")) + .isInstanceOf(IllegalArgumentException.class) + .hasMessageContaining("Missing value kind code"); + assertThatThrownBy(() -> ValueStoreWalValueKind.fromCode(null)) + .isInstanceOf(IllegalArgumentException.class) + .hasMessageContaining("Missing value kind code"); + } +} diff --git a/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/WalSyncBootstrapOnOpenTest.java b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/WalSyncBootstrapOnOpenTest.java new file mode 100644 index 00000000000..965d46a5933 --- /dev/null +++ b/core/sail/nativerdf/src/test/java/org/eclipse/rdf4j/sail/nativerdf/wal/WalSyncBootstrapOnOpenTest.java @@ -0,0 +1,86 @@ +/******************************************************************************* + * Copyright (c) 2025 Eclipse RDF4J contributors. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Distribution License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/org/documents/edl-v10.php. + * + * SPDX-License-Identifier: BSD-3-Clause + *******************************************************************************/ +package org.eclipse.rdf4j.sail.nativerdf.wal; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.File; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.LinkedHashMap; +import java.util.Map; +import java.util.UUID; + +import org.eclipse.rdf4j.common.io.ByteArrayUtil; +import org.eclipse.rdf4j.model.IRI; +import org.eclipse.rdf4j.model.impl.SimpleValueFactory; +import org.eclipse.rdf4j.sail.nativerdf.ValueStore; +import org.eclipse.rdf4j.sail.nativerdf.datastore.DataStore; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +/** + * Verifies that when configured with syncBootstrapOnOpen=true, the ValueStore rebuilds the WAL synchronously during + * open before returning, so the WAL already contains entries for existing values. + */ +class WalSyncBootstrapOnOpenTest { + + @TempDir + Path tempDir; + + @Test + void bootstrapSynchronousOnOpen() throws Exception { + // Arrange: create a ValueStore dictionary without WAL + Path dataDir = tempDir.resolve("store"); + Files.createDirectories(dataDir); + try (DataStore ds = new DataStore(new File(dataDir.toString()), "values", false)) { + // Store a namespace and an IRI value + int nsId = ds.storeData("http://example.org/".getBytes(StandardCharsets.UTF_8)); + IRI iri = SimpleValueFactory.getInstance().createIRI("http://example.org/x"); + byte[] local = iri.getLocalName().getBytes(StandardCharsets.UTF_8); + byte[] iriBytes = new byte[1 + 4 + local.length]; + iriBytes[0] = 0x1; + ByteArrayUtil.putInt(nsId, iriBytes, 1); + ByteArrayUtil.put(local, iriBytes, 5); + ds.storeData(iriBytes); + ds.sync(); + } + + // Act: open ValueStore with WAL configured to synchronous bootstrap + Path walDir = dataDir.resolve(ValueStoreWalConfig.DEFAULT_DIRECTORY_NAME); + ValueStoreWalConfig cfg = ValueStoreWalConfig.builder() + .walDirectory(walDir) + .storeUuid(UUID.randomUUID().toString()) + .syncBootstrapOnOpen(true) + .build(); + + try (ValueStoreWAL wal = ValueStoreWAL.open(cfg); + ValueStore vs = new ValueStore(new File(dataDir.toString()), false, + ValueStore.VALUE_CACHE_SIZE, ValueStore.VALUE_ID_CACHE_SIZE, + ValueStore.NAMESPACE_CACHE_SIZE, ValueStore.NAMESPACE_ID_CACHE_SIZE, wal)) { + // Upon return, bootstrap should be complete and WAL should contain records for existing values + } + + // Assert: WAL contains at least the namespace and the IRI records + Map dictionary; + try (ValueStoreWalReader reader = ValueStoreWalReader.open(cfg)) { + ValueStoreWalRecovery recovery = new ValueStoreWalRecovery(); + dictionary = new LinkedHashMap<>(recovery.replay(reader)); + } + assertThat(dictionary).isNotEmpty(); + assertThat(dictionary.values().stream().anyMatch(r -> r.valueKind() == ValueStoreWalValueKind.NAMESPACE)) + .isTrue(); + assertThat(dictionary.values() + .stream() + .anyMatch(r -> r.valueKind() == ValueStoreWalValueKind.IRI && r.lexical().endsWith("/x"))).isTrue(); + } +} diff --git a/pom.xml b/pom.xml index 1d2f315909a..97652247dbd 100644 --- a/pom.xml +++ b/pom.xml @@ -842,7 +842,7 @@ 3.5.4 - @{argLine} -Xmx2G + @{argLine} -Xmx4G @@ -853,7 +853,7 @@ 1 false - @{argLine} -Xmx2G + @{argLine} -Xmx4G **/*IT.java diff --git a/site/content/documentation/reference/configuration.md b/site/content/documentation/reference/configuration.md index 5b5f8f96f45..93e0e121508 100644 --- a/site/content/documentation/reference/configuration.md +++ b/site/content/documentation/reference/configuration.md @@ -271,6 +271,79 @@ The native store automatically creates/drops indexes upon (re)initialization, so ]. ``` +##### ValueStore write-ahead log + +The NativeStore maintains a write-ahead log (WAL) for its value dictionary so that newly minted IRIs, blank nodes, literals and namespaces can be recovered independently from the main on-disk `values*` files. The WAL lives in a `value-store-wal/` directory under the repository data dir and is protected by a lock file to prevent concurrent writers. + +###### When the WAL is active +- Enabled automatically for writable data directories. Read-only deployments continue without a WAL. +- Existing repositories that upgrade to a WAL will have the log bootstrapped from their current value files. By default this bootstrap happens asynchronously in the background so startup is not blocked; set `config:native.walSyncBootstrapOnOpen true` if you prefer to wait for a complete log before accepting new writes. +- Clearing a store via the API purges all WAL segments so that deleted values cannot be resurrected during later recovery. + +###### What the WAL records (and what it does not) +- Records every newly minted value together with its internal ID, lexical form, language/datatype metadata, and a CRC32C hash. Segments are append-only and rotated once they reach `config:native.walMaxSegmentBytes` (128 MiB by default); completed segments are gzip-compressed with an integrity summary frame. +- The WAL does **not** track statement inserts/removals or other file sets (triple indexes, context store, namespace store). These still rely on the existing NativeStore commit process. +- The log is a durability and recovery aid: the regular `values*.dat` files remain the primary source of truth. If you remove the WAL you lose the ability to rebuild the value dictionary from the log, but the store continues to operate. + +###### Durability policies and performance +- The background WAL writer batches records in a direct ByteBuffer (`config:native.walBatchBufferBytes`, default 128 KiB) and drains a bounded queue (`config:native.walQueueCapacity`, default 16,384 records). Producers spin briefly and then block when the queue is full, so sustained high write rates should tune these parameters instead of disabling the WAL. +- `config:native.walSyncPolicy` controls when segments are forced to disk: + - `COMMIT` waits for the store's commit path to call `awaitWalDurable`, so the WAL is forced in sync with transaction commits. + - `INTERVAL` (default) forces at most every `config:native.walSyncIntervalMillis` (default 1000 ms) even if no commit is pending (useful for long-running bulk loads that rarely commit). It trades durability for throughput and is **not ACID-safe**: values committed between fsyncs may be lost if the process or host crashes. + - `ALWAYS` fsyncs after every frame for the lowest data-loss window at the cost of throughput. +- A small idle poll (`config:native.walIdlePollIntervalMillis`, default 100 ms) keeps latency low without busy-waiting when the queue is empty. + +###### Recovery options +- Keep `config:native.walSyncBootstrapOnOpen` at its default (`false`) for large stores that favour fast restarts. Switch it on to guarantee the WAL contains the complete dictionary before accepting traffic (helpful when you move a data directory between hosts). +- Enable `config:native.walAutoRecoverOnOpen true` to have the ValueStore rebuild missing or empty `values*` files from the WAL during startup. Recovery only runs when the WAL dictionary is complete and contiguous; the store logs a warning and skips recovery if segments are missing or truncated. +- Diagnostic logging for WAL recovery can be tuned with the JVM system property `-Dorg.eclipse.rdf4j.sail.nativerdf.valuestorewal.recoveryLog=trace|debug|off`. +- Advanced administrators can inspect the log with the utility classes in `org.eclipse.rdf4j.sail.nativerdf.wal` (for example `ValueStoreWalReader` and `ValueStoreWalSearch`) to verify entries or extract lost value metadata. + +###### Configuration summary +- `config:native.walMaxSegmentBytes` → rotate segments sooner than the 128 MiB default if you prefer smaller compressed files. +- `config:native.walQueueCapacity` / `config:native.walBatchBufferBytes` → increase when bulk loading outpaces the background writer. +- `config:native.walDirectoryName` → place the WAL on a dedicated volume (the path is resolved inside the data dir). +- `config:native.walSyncPolicy`, `config:native.walSyncIntervalMillis`, `config:native.walIdlePollIntervalMillis` → tune durability/latency trade-offs. +- `config:native.walSyncBootstrapOnOpen`, `config:native.walAutoRecoverOnOpen` → control bootstrap timing and automatic rebuild behaviour. +- `config:native.walEnabled false` → turn the WAL off entirely if you need legacy behaviour or are operating on ephemeral data; the store will log that value repairs can no longer be replayed from the log. + +###### Example configuration (Turtle) + +```turtle +@prefix config: . + +[] a config:Repository ; + config:rep.id "native-with-wal" ; + config:rep.impl [ + config:rep.type "openrdf:SailRepository" ; + config:sail.impl [ + config:sail.type "openrdf:NativeStore" ; + config:native.walSyncPolicy "INTERVAL" ; + config:native.walSyncIntervalMillis 5 ; + config:native.walMaxSegmentBytes 268435456 ; # 256 MiB + config:native.walQueueCapacity 524288 ; + config:native.walSyncBootstrapOnOpen true ; + config:native.walAutoRecoverOnOpen true ; + config:native.walEnabled true + ] + ]. +``` + +###### Programmatic setup (Java) + +```java +NativeStore store = new NativeStore(dataDir); +store.setWalSyncPolicy(ValueStoreWalConfig.SyncPolicy.INTERVAL); +store.setWalSyncIntervalMillis(5); +store.setWalMaxSegmentBytes(256L * 1024 * 1024); +store.setWalQueueCapacity(524_288); +store.setWalSyncBootstrapOnOpen(true); +store.setWalAutoRecoverOnOpen(true); +store.setWalEnabled(true); // or false to disable the WAL entirely +``` + +When copying or backing up a repository, include the entire `value-store-wal/` directory (lock file, `store.uuid`, and `wal-*.v1[.gz]` segments) alongside the main NativeStore data files to preserve the WAL history. + #### Elasticsearch Store The Elasticsearch Store is an RDF4J database that persists all data directly in Elasticsearch (not to be confused with the Elasticsearch Fulltext Search Sail, which is an adapter Sail implementation to provided full-text search indexing on top of other RDF databases). Its `config:sail.type` value is `"rdf4j:ElasticsearchStore"`. @@ -482,4 +555,3 @@ The fully rewritten configuration looks like this: ``` Note that we have not (yet) renamed the type identifier literals `openrdf:SailRepository` and `openrdf:NativeStore`. For more details we refer you to the {{< javadoc "CONFIG javadoc" "model/vocabulary/CONFIG.html" >}}. -