Feature/ingest v2 #439

ag-ramachandran · 2025-09-09T05:59:32Z

Initial work on the IngestV2 API

Copilot

Pull Request Overview

This PR introduces the initial work on the IngestV2 API, implementing a new Kotlin-based ingestion service alongside the existing Java implementation. The major change is upgrading the project from Java 8 to Java 11 and updating various dependencies to support the new ingest-v2 module.

Project upgraded from Java 8 to Java 11 with dependency version updates
New ingest-v2 module written in Kotlin with modern async capabilities
Centralized dependency management moved to root POM for consistency

Reviewed Changes

Copilot reviewed 29 out of 29 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
pom.xml	Upgraded Java version, updated dependencies, added ingest-v2 module, centralized dependency management
ingest/pom.xml	Removed duplicate dependency management, cleaned up version specifications
data/pom.xml	Removed duplicate dependency management, cleaned up version specifications
ingest-v2/pom.xml	Complete new module configuration with Kotlin, Ktor, and OpenAPI generation
ingest-v2/src/main/resources/openapi.yaml	OpenAPI specification for the new ingest REST API
ingest-v2/src/main/kotlin/...	Core Kotlin implementation files for authentication, retry policies, data sources, and API clients
ingest-v2/src/test/kotlin/...	Unit tests for retry policies and configuration API

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

...main/kotlin/com/microsoft/azure/kusto/ingest/v2/common/auth/AzCliTokenCredentialsProvider.kt

ingest-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/source/BlobSource.kt

ingest-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/source/LocalSource.kt

ingest-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/common/utils/PathUtils.kt

github-actions · 2025-09-09T06:00:40Z

Test Results

529 tests ±0 520 ✅ ±0 3m 41s ⏱️ +49s
31 suites ±0 9 💤 ±0
31 files ±0 0 ❌ ±0

Results for commit 31ff143. ± Comparison against base commit 353cd2c.

♻️ This comment has been updated with latest results.

AsafMah · 2025-10-30T09:59:07Z

pom.xml

        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
-        <java.version>1.8</java.version>
-        <azure-bom-version>1.2.28</azure-bom-version>
+        <java.version>11</java.version>


We discussed waiting with this until another PR, right?

Hello @AsafMah , The overall upgrade can be on another PR. But for ingest-v2 as a seperate module we can use Java 11. The changes to code will be very large if we cannot use 11 and fall back to JDK8.

For starters, kotlinx_serialization is not supported on Java-8

I think I understand the confusion, apologies for it, What was intended was, when we update kusto-data and other modules that can be another PR using open-rewrite which is automated
https://docs.openrewrite.org/

ingest-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/common/utils/PathUtils.kt

ingest-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/source/IngestionSource.kt

ingest-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/container/UploadContainerBase.kt

ingest-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/container/BlobUploadContainer.kt

ingest-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/source/DataSourceFormat.kt

* Address review comments

Copilot

Pull Request Overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 5 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-10-31T13:16:14Z

ingest-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/common/ConfigurationCache.kt

+import kotlin.text.compareTo
+


The import kotlin.text.compareTo is unused in this file. Consider removing it to keep imports clean.

Suggested change

import kotlin.text.compareTo

Copilot · 2025-10-31T13:16:14Z

ingest-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/source/IngestionSource.kt

+
+import com.microsoft.azure.kusto.ingest.v2.common.utils.PathUtils
+import com.microsoft.azure.kusto.ingest.v2.models.Format
+import java.lang.AutoCloseable


Use kotlin.AutoCloseable instead of java.lang.AutoCloseable for better Kotlin interoperability. Kotlin provides its own AutoCloseable interface as a typealias.

Suggested change

import java.lang.AutoCloseable

import kotlin.AutoCloseable

Copilot · 2025-10-31T13:16:15Z

ingest-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/common/ConfigurationCache.kt

+package com.microsoft.azure.kusto.ingest.v2.common
+
+import com.microsoft.azure.kusto.ingest.v2.models.ConfigurationResponse
+import java.lang.AutoCloseable


Use kotlin.AutoCloseable instead of java.lang.AutoCloseable for better Kotlin interoperability. Kotlin provides its own AutoCloseable interface as a typealias.

Suggested change

import java.lang.AutoCloseable

Copilot · 2025-10-31T13:16:15Z

ingest-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/common/ConfigurationCache.kt

+    private val configurationProvider: suspend () -> ConfigurationResponse,
+) : ConfigurationCache {
+    @Volatile private var cachedConfiguration: ConfigurationResponse? = null
+    private var lastRefreshTime: Long = 0


The field lastRefreshTime is not marked as @Volatile but is accessed from multiple threads (read outside synchronized block, written inside). This could lead to visibility issues where threads may see stale values. Mark it as @Volatile for proper thread visibility.

Suggested change

private var lastRefreshTime: Long = 0

@Volatile private var lastRefreshTime: Long = 0

Copilot · 2025-10-31T13:16:15Z

ingest-v2/pom.xml

+                    <compilerPlugins>
+                        <plugin>kotlinx-serialization</plugin>
+                    </compilerPlugins>
+                    <jvmTarget>1.8</jvmTarget>


The Kotlin compiler is targeting JVM 1.8, but the parent pom specifies Java 11 as the minimum version. For consistency and to take advantage of Java 11 features, consider updating jvmTarget to 11.

Suggested change

<jvmTarget>1.8</jvmTarget>

<jvmTarget>11</jvmTarget>

mayamarom10 · 2025-11-17T10:00:13Z

ingest-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/ConfigurationApiWrapper.kt

+    override val dmUrl: String,
+    override val tokenCredential: TokenCredential,
+    override val skipSecurityChecks: Boolean = false,
+) : KustoBaseApiClient(dmUrl, tokenCredential, skipSecurityChecks) {


Are we consistently using this naming here (Data management URL)? In the portal it's called 'Data Ingestion URI' - might be a good idea to align with that for clarity.

mayamarom10 · 2025-11-17T10:02:56Z

ingest-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/common/RetryPolicyExtensions.kt

+
+            when (decision) {
+                RetryDecision.Throw -> {
+                    tracer?.invoke(


Why would the tracer be null? And if it's null do we not log the errors at all?

mayamarom10 · 2025-11-17T10:57:06Z

ingest-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/source/LocalSource.kt

+    val leaveOpen: Boolean,
+    override val compressionType: CompressionType = CompressionType.NONE,
+    val baseName: String? = null,
+    override val sourceId: UUID = UUID.randomUUID(),


Do we prefer a random UUID? In C# SDK we use null

* Add IngestionSource Blob type and QueuedIngest * Merge code with IngestV2 branch * Reorganize code and remove DTO's * * Fix some review comments --------- Co-authored-by: ag-ramachandran <ramacg@microsoft.com>

* * Add boilerplate * Merge code with IngestV2 branch * * Add support for non-blob sources * * Add support for non-blob sources * * Add support for File and Stream uploads * managed streaming ingest client * * Minor edits - Make constants for ManagedStreamingIngestPolicy * Add a method called size for determining size in the AbstractSourceInfo interface --------- Co-authored-by: Ramachandran A G <ramacg@microsoft.com> * * Fix some formatting * * Remove unused config --------- Co-authored-by: ag-ramachandran <ramacg@microsoft.com>

ag-ramachandran requested review from AsafMah, Copilot and tanmaya-panda1 September 9, 2025 05:59

ag-ramachandran had a problem deploying to build September 9, 2025 05:59 — with GitHub Actions Failure

ag-ramachandran had a problem deploying to build September 9, 2025 05:59 — with GitHub Actions Error

Copilot AI reviewed Sep 9, 2025

View reviewed changes

ag-ramachandran had a problem deploying to build September 9, 2025 10:10 — with GitHub Actions Error

ag-ramachandran had a problem deploying to build September 9, 2025 10:10 — with GitHub Actions Failure

ag-ramachandran had a problem deploying to build September 9, 2025 10:10 — with GitHub Actions Error

ag-ramachandran had a problem deploying to build September 9, 2025 16:39 — with GitHub Actions Error

ag-ramachandran had a problem deploying to build September 9, 2025 16:39 — with GitHub Actions Failure

ag-ramachandran had a problem deploying to build September 9, 2025 16:39 — with GitHub Actions Error

ag-ramachandran had a problem deploying to build September 10, 2025 01:24 — with GitHub Actions Error

ag-ramachandran had a problem deploying to build September 10, 2025 01:24 — with GitHub Actions Failure

ag-ramachandran had a problem deploying to build September 10, 2025 01:24 — with GitHub Actions Error

ag-ramachandran temporarily deployed to build September 10, 2025 01:57 — with GitHub Actions Inactive

ag-ramachandran temporarily deployed to build September 10, 2025 12:15 — with GitHub Actions Inactive

ag-ramachandran temporarily deployed to build September 10, 2025 12:20 — with GitHub Actions Inactive

AsafMah marked this pull request as ready for review September 18, 2025 09:12

* Rename retry data class

8cc94f5

ag-ramachandran temporarily deployed to build October 16, 2025 05:32 — with GitHub Actions Inactive

AsafMah requested changes Oct 30, 2025

View reviewed changes

* Remove unused classes

87f4fed

* Address review comments

ag-ramachandran temporarily deployed to build October 30, 2025 11:04 — with GitHub Actions Inactive

* Remove unused classes

dca3002

ag-ramachandran temporarily deployed to build October 30, 2025 11:06 — with GitHub Actions Inactive

* Fix comment on substring chaining

31ff143

ag-ramachandran temporarily deployed to build October 31, 2025 05:09 — with GitHub Actions Inactive

ag-ramachandran requested review from AsafMah, Copilot, nir-boger and yogilad October 31, 2025 13:11

Copilot AI reviewed Oct 31, 2025

View reviewed changes

ag-ramachandran requested a review from mayamarom10 November 12, 2025 09:42

mayamarom10 reviewed Nov 17, 2025

View reviewed changes

tanmaya-panda1 and others added 4 commits December 2, 2025 15:08

Feature/add ingestion source blob (#440)

b484e73

* Add IngestionSource Blob type and QueuedIngest * Merge code with IngestV2 branch * Reorganize code and remove DTO's * * Fix some review comments --------- Co-authored-by: ag-ramachandran <ramacg@microsoft.com>

disabled serialized execution

dc308e1

optimized QueuedIngestionClientTests execution

533cd35

	private var lastRefreshTime: Long = 0
	@Volatile private var lastRefreshTime: Long = 0

Feature/ingest v2 #439

Are you sure you want to change the base?

Feature/ingest v2 #439

Uh oh!

Conversation

ag-ramachandran commented Sep 9, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

Uh oh!

AsafMah Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

ag-ramachandran Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

ag-ramachandran Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Copilot AI Oct 31, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 31, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 31, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 31, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 31, 2025

Choose a reason for hiding this comment

Uh oh!

mayamarom10 Nov 17, 2025

Choose a reason for hiding this comment

Uh oh!

mayamarom10 Nov 17, 2025

Choose a reason for hiding this comment

Uh oh!

mayamarom10 Nov 17, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

github-actions bot commented Sep 9, 2025 •

edited

Loading

ag-ramachandran Oct 30, 2025 •

edited

Loading