Feature/ingest v2 #439
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,38 @@ | ||
| # ingest-v2 | ||
|
|
||
| This project was created using the [Ktor Project Generator](https://start.ktor.io). | ||
|
|
||
| Here are some useful links to get you started: | ||
|
|
||
| - [Ktor Documentation](https://ktor.io/docs/home.html) | ||
| - [Ktor GitHub page](https://github.com/ktorio/ktor) | ||
| - The [Ktor Slack chat](https://app.slack.com/client/T09229ZC6/C0A974TJ9). You'll need to [request an invite](https://surveys.jetbrains.com/s3/kotlin-slack-sign-up) to join. | ||
|
|
||
| ## Features | ||
|
|
||
| Here's a list of features included in this project: | ||
|
|
||
| | Name | Description | | ||
| |------------------------------------------------------------------------|------------------------------------------------------------------------------------| | ||
| | [Content Negotiation](https://start.ktor.io/p/content-negotiation) | Provides automatic content conversion according to Content-Type and Accept headers | | ||
| | [Routing](https://start.ktor.io/p/routing) | Provides a structured routing DSL | | ||
| | [kotlinx.serialization](https://start.ktor.io/p/kotlinx-serialization) | Handles JSON serialization using kotlinx.serialization library | | ||
| | [AsyncAPI](https://start.ktor.io/p/asyncapi) | Generates and serves AsyncAPI documentation | | ||
|
|
||
| ## Building & Running | ||
|
|
||
| To build or run the project, use one of the following tasks: | ||
|
|
||
| | Task | Description | | ||
| |--------------------------------------------------------------|-------------------| | ||
| | `mvn test` | Run the tests | | ||
| | `mvn package` | Build the project | | ||
| | `java -jar target/ingest-v2-0.0.1-jar-with-dependencies.jar` | Run the server | | ||
|
|
||
| If the server starts successfully, you'll see the following output: | ||
|
|
||
| ``` | ||
| 2024-12-04 14:32:45.584 [main] INFO Application - Application started in 0.303 seconds. | ||
| 2024-12-04 14:32:45.682 [main] INFO Application - Responding at http://0.0.0.0:8080 | ||
| ``` | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,215 @@ | ||||||
| <?xml version="1.0" encoding="UTF-8"?> | ||||||
| <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" | ||||||
| xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> | ||||||
| <modelVersion>4.0.0</modelVersion> | ||||||
| <artifactId>ingest-v2</artifactId> | ||||||
| <version>${revision}</version> | ||||||
| <name>ingest-v2</name> | ||||||
| <description>ingest-v2</description> | ||||||
| <properties> | ||||||
| <kotlin.code.style>official</kotlin.code.style> | ||||||
| <kotlin.version>2.2.20</kotlin.version> | ||||||
| <ktor.version>3.3.0</ktor.version> | ||||||
| <ktor.async.api.version>3.1.1</ktor.async.api.version> | ||||||
| <kotlinx.coroutines.debug.version>1.10.2</kotlinx.coroutines.debug.version> | ||||||
| <logback.version>1.4.14</logback.version> | ||||||
| <junit.version>5.10.0</junit.version> | ||||||
| <openapi.generator.version>7.15.0</openapi.generator.version> | ||||||
| <slf4j.version>2.0.9</slf4j.version> | ||||||
| <spotless.version>2.46.1</spotless.version> | ||||||
| <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> | ||||||
| <kotlin.compiler.incremental>true</kotlin.compiler.incremental> | ||||||
| </properties> | ||||||
| <parent> | ||||||
| <artifactId>kusto-client</artifactId> | ||||||
| <groupId>com.microsoft.azure.kusto</groupId> | ||||||
| <!--suppress MavenPropertyInParent --> | ||||||
| <version>${revision}</version> | ||||||
| </parent> | ||||||
| <dependencies> | ||||||
| <dependency> | ||||||
| <groupId>io.ktor</groupId> | ||||||
| <artifactId>ktor-client-auth-jvm</artifactId> | ||||||
| <version>${ktor.version}</version> | ||||||
| </dependency> | ||||||
| <dependency> | ||||||
| <groupId>io.ktor</groupId> | ||||||
| <artifactId>ktor-client-content-negotiation-jvm</artifactId> | ||||||
| <version>${ktor.version}</version> | ||||||
| </dependency> | ||||||
| <dependency> | ||||||
| <groupId>io.ktor</groupId> | ||||||
| <artifactId>ktor-serialization-kotlinx-json-jvm</artifactId> | ||||||
| <version>${ktor.version}</version> | ||||||
| </dependency> | ||||||
| <dependency> | ||||||
| <groupId>io.ktor</groupId> | ||||||
| <artifactId>ktor-client-java-jvm</artifactId> | ||||||
| <version>${ktor.version}</version> | ||||||
| </dependency> | ||||||
| <dependency> | ||||||
| <groupId>org.slf4j</groupId> | ||||||
| <artifactId>slf4j-simple</artifactId> | ||||||
| <version>${slf4j.version}</version> | ||||||
| </dependency> | ||||||
| <dependency> | ||||||
| <groupId>com.azure</groupId> | ||||||
| <artifactId>azure-identity</artifactId> | ||||||
| </dependency> | ||||||
| <dependency> | ||||||
| <groupId>org.jetbrains.kotlin</groupId> | ||||||
| <artifactId>kotlin-test-junit5</artifactId> | ||||||
| <version>${kotlin.version}</version> | ||||||
| <scope>test</scope> | ||||||
| </dependency> | ||||||
| <dependency> | ||||||
| <groupId>org.jetbrains.kotlinx</groupId> | ||||||
| <artifactId>kotlinx-coroutines-debug</artifactId> | ||||||
| <version>${kotlinx.coroutines.debug.version}</version> | ||||||
| <scope>test</scope> | ||||||
| </dependency> | ||||||
| <dependency> | ||||||
| <groupId>io.mockk</groupId> | ||||||
| <artifactId>mockk-jvm</artifactId> | ||||||
| <version>1.14.5</version> | ||||||
| <scope>test</scope> | ||||||
| </dependency> | ||||||
| <dependency> | ||||||
| <groupId>org.junit.jupiter</groupId> | ||||||
| <artifactId>junit-jupiter-params</artifactId> | ||||||
| <version>${junit.version}</version> | ||||||
| <scope>test</scope> | ||||||
| </dependency> | ||||||
| </dependencies> | ||||||
| <build> | ||||||
| <sourceDirectory>${project.basedir}/src/main/kotlin</sourceDirectory> | ||||||
| <testSourceDirectory>${project.basedir}/src/test/kotlin</testSourceDirectory> | ||||||
| <resources> | ||||||
| <resource> | ||||||
| <directory>${project.basedir}/src/main/resources</directory> | ||||||
| </resource> | ||||||
| </resources> | ||||||
| <plugins> | ||||||
| <plugin> | ||||||
| <groupId>org.jetbrains.kotlin</groupId> | ||||||
| <artifactId>kotlin-maven-plugin</artifactId> | ||||||
| <version>${kotlin.version}</version> | ||||||
| <executions> | ||||||
| <execution> | ||||||
| <id>compile</id> | ||||||
| <phase>compile</phase> | ||||||
| <goals> | ||||||
| <goal>compile</goal> | ||||||
| </goals> | ||||||
| <configuration> | ||||||
| <sourceDirs> | ||||||
| <sourceDir>${project.basedir}/src/main/kotlin</sourceDir> | ||||||
| <sourceDir>${project.build.directory}/generated-sources/openapi/src/main/kotlin</sourceDir> | ||||||
| </sourceDirs> | ||||||
| </configuration> | ||||||
| </execution> | ||||||
| <execution> | ||||||
| <id>test-compile</id> | ||||||
| <phase>test-compile</phase> | ||||||
| <goals> | ||||||
| <goal>test-compile</goal> | ||||||
| </goals> | ||||||
| </execution> | ||||||
| </executions> | ||||||
| <configuration> | ||||||
| <compilerPlugins> | ||||||
| <plugin>kotlinx-serialization</plugin> | ||||||
| </compilerPlugins> | ||||||
| <jvmTarget>1.8</jvmTarget> | ||||||
|
||||||
| <jvmTarget>1.8</jvmTarget> | |
| <jvmTarget>11</jvmTarget> |
Contributor
There was a problem hiding this comment.
Generating doesn't work for me at all - I'm getting this error:
Execution default-cli of goal org.openapitools:openapi-generator-maven-plugin:7.15.0:generate failed: Unable to load the mojo 'generate' in the plugin 'org.openapitools:openapi-generator-maven-plugin:7.15.0' due to an API incompatibility: org.codehaus.plexus.component.repository.exception.ComponentLookupException: org/openapitools/codegen/plugin/CodeGenMojo has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0
Contributor
There was a problem hiding this comment.
Which indicates a problem with using java 8 - using a later version works. We need to support java 8.
Contributor
There was a problem hiding this comment.
Getting the following warnings:
Unknown type found in the schema: long. To map it, please use the schema mapping option (e.g. --schema-mappings in CLI)
Unknown type found in the schema: long. To map it, please use the schema mapping option (e.g. --schema-mappings in CLI)
Unknown type found in the schema: long. To map it, please use the schema mapping option (e.g. --schema-mappings in CLI)
Empty operationId found for path: post /v1/rest/ingestion/queued/{database}/{table}. Renamed to auto-generated operationId: v1RestIngestionQueuedDatabaseTablePost
Empty operationId found for path: get /v1/rest/ingestion/configuration. Renamed to auto-generated operationId: v1RestIngestionConfigurationGet
Empty operationId found for path: get /v1/rest/ingestion/queued/{database}/{table}/{operationId}. Renamed to auto-generated operationId: v1RestIngestionQueuedDatabaseTableOperationIdGet
Empty operationId found for path: post /v1/rest/ingest/{database}/{table}. Renamed to auto-generated operationId: v1RestIngestDatabaseTablePost
Contributor
Author
There was a problem hiding this comment.
You're right, this is because we are using 11.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| // Copyright (c) Microsoft Corporation. | ||
| // Licensed under the MIT License. | ||
| package com.microsoft.azure.kusto.ingest.v2 | ||
|
|
||
| import com.azure.core.credential.TokenCredential | ||
| import com.microsoft.azure.kusto.ingest.v2.apis.DefaultApi | ||
| import com.microsoft.azure.kusto.ingest.v2.common.exceptions.IngestException | ||
| import com.microsoft.azure.kusto.ingest.v2.infrastructure.HttpResponse | ||
| import com.microsoft.azure.kusto.ingest.v2.models.ConfigurationResponse | ||
| import org.slf4j.LoggerFactory | ||
|
|
||
| class ConfigurationApiWrapper( | ||
|
||
| override val dmUrl: String, | ||
| override val tokenCredential: TokenCredential, | ||
| override val skipSecurityChecks: Boolean = false, | ||
| ) : KustoBaseApiClient(dmUrl, tokenCredential, skipSecurityChecks) { | ||
|
||
| private val logger = | ||
| LoggerFactory.getLogger(ConfigurationApiWrapper::class.java) | ||
| private val baseUrl = "$dmUrl/v1/rest/ingestion/configuration" | ||
| private val api: DefaultApi = | ||
| DefaultApi(baseUrl = dmUrl, httpClientConfig = setupConfig) | ||
|
|
||
| suspend fun getConfigurationDetails(): ConfigurationResponse { | ||
| val configurationHttpResponse: HttpResponse<ConfigurationResponse> = | ||
| api.getIngestConfiguration() | ||
| if (configurationHttpResponse.success) { | ||
| logger.info( | ||
| "Successfully retrieved configuration details from $dmUrl with status: ${configurationHttpResponse.status}", | ||
| ) | ||
| logger.debug( | ||
| "Configuration details: {}", | ||
| configurationHttpResponse.body(), | ||
| ) | ||
| return configurationHttpResponse.body() | ||
| } else { | ||
| logger.error( | ||
| "Failed to retrieve configuration details from $baseUrl. Status: ${configurationHttpResponse.status}, " + | ||
| "Body: ${configurationHttpResponse.body()}", | ||
| ) | ||
| throw IngestException( | ||
| "Failed to retrieve configuration details from $baseUrl. Status: ${configurationHttpResponse.status}, " + | ||
| "Body: ${configurationHttpResponse.body()}", | ||
| ) | ||
| } | ||
| } | ||
| } | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why did you remove java 8?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
initial thought:
Some internal projects (spark specifically) were using Java8. However newer distributions start at 11 (and are moving to 17 as well).