Use Quarkus Rest Client instead of splunk-library-javalogging

[vietk]

Quarkus logging splunk is using splunk-library-logging third party to reach the Splunk Event collector.

The idea is to replace this dependency by a Quarkus dedicated Rest Client for :

• Performance/troubleshotting reason (Performance issue when using json mode splunk/splunk-library-javalogging#257, Potential memory issues when slow HEC responses #255)
• Unifying the HTTP client (Splunk's Library is using ok http)
• Slow release cycle/ project activity
• Event Collector REST API is public - https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/HECRESTendpoints
• Unnecessary logging system support provided (logback/log4j ... )
• No defined metrics and HTTP Client metrics produced

[jcjveraa]

Hi @vietk - any progress on picking this up?
As you may guess from the various issues I'm posting, I'm working on getting an application to production that uses this extension for logging. When testing the 'retries' option, I ran into what this PR of the splunk library tries to solve - retries are very poorly implemented for no clear reason, the library will only retry something that already had a 200 HTTP response.
splunk/splunk-library-javalogging#287

I cannot understand what the original author of this comment thought - it would seem pretty clear to me that a 500-range error should be retried, the official docs clearly state that a 503 could be expected, and in our organisation we have indeed seen error 503's when the HEC queues were full.
https://github.com/splunk/splunk-library-javalogging/blob/70f80fa60d2098e81f5f819059964d9304b5f1d5/src/main/java/com/splunk/logging/HttpEventCollectorResendMiddleware.java#L69
https://docs.splunk.com/Documentation/Splunk/9.4.0/RESTREF/RESTinput#HTTP_status_codes

[vietk]

Hello

No real progress on our hand, because it's a bit of a big bang.

However to answer your specific question there's now a specific entrypoint in the library that has been introduced by the PR #260.
It's now possible to implement your own retry mechanism by providing your own HttpSenderMiddleware.
A proposal would be to decorate the default resend middlware and provide an implementation that is capable of retry on the error of your choice by overriding completed

I hope it answers your question.

Regards
Kevin

[jcjveraa]

Thanks for the quick response @vietk! I have 'naively' copied over the complete class as suggested in splunk/splunk-library-javalogging#287 as it does what we need, and it 'just works' when injecting it as a middleware via then config. Trying to extend/decorate it is more complex as the Callback which contains the complete method you mentioned is a private inner class.

I had to hardcode the 'max retries' value, as I see no way of passing that as an argument when loading the class via application.properties.

Regardless I would encourage moving away from the official splunk-logger library as I see no benefit and apparently it's not maintained. Do you already have a (mental) list of things that should be implemented in a RestClient solution? I can have a chat with my team manager to see if I would be allowed to contribute on company time.

[rquinio1A]

An alternative to consider is the new OpenTelemetry logging extension (https://quarkus.io/guides/opentelemetry-logging) and using a Splunk backend in the opentelemetry collector (which is basically a REST client of HEC, but written in Go).

However this may be too early:

• the Quarkus extension is in technology preview and not yet in a Quarkus LTS.
• the tuning of opentelemetry-collector for logging traffic, which is typically higher than tracing.

[vietk]

Do you already have a (mental) list of things that should be implemented in a RestClient solution

Naïvely speaking, mostly what is already present in HttpEventCollectorSender (serialization, batching, Metadatas ...).
The client communication part implemented would be implemented with RestClient (based on Splunk Rest API) and FaultTolerance.

We saw that when the HEC endpoint is slow (at the limit of reaching timeout) there is potential OOME because we suspect the splunk's client library to not limit the events in memory, even if put in place in the library), then the enqueing method should be capped with a rate limiting process (could potentially be implemented using microprofile BulkHead).