Skip to content

Commit 7a29338

Browse files
committed
Save HTTP connection across requests
This influxdb client is not using persistent connections. This causes a significant performance loss. Below is a benchmark script that writes a single point to influxdb. Running it takes 44 seconds of time: $ ruby -Ilib t.rb Writing 1000 points user system total real write_points 19.270388 1.394931 20.665319 ( 44.622170) This works out to a write speed of about 22 points per second, or about 50ms per point. This is: * Pretty slow. If you are generating points quickly maximum resolution of 50ms is not great. A user may lose data because they cannot sample quickly enough. * Pretty inefficient. For each point written the client library must establish a TCP connection, establish a TLS session, finally write some data. Any data written may be restricted by the TCP slow-start windowing algorithm. Much more ruby code must be run, almost half the time spent is in the "user" category, time that could be doing anything else in a ruby application, or time that could be used by other processes. This commit caches HTTP connections across requests. Running the same benchmark takes 4.26 seconds: $ ruby -Ilib t.rb Writing 1000 points user system total real write_points 0.551663 0.084603 0.636266 ( 4.261201) This works out to a speed of 234 points per second, or about 5ms per point. Writing points now no longer need to recreate a TCP connection, renegotiate a TLS session, or be held up by TCP window sizing limitations. This is much more efficient in terms of CPU time used per point, instead of ~46% of time taken occurring in the "user" category, now only 13% of time is in "user". The balance of the time is now spent waiting for IO to complete. Benchmark script: require "benchmark" require "influxdb" n = Integer ENV["N"] rescue 1_000 influxdb = InfluxDB::Client.new url: ENV["INFLUX_URL"], username: ENV["INFLUX_USER"], password: ENV["INFLUX_PASS"], time_precision: "u" def write_point counter, influxdb points = [ { series: "test", values: { counter: counter, }, }, ] influxdb.write_points points end puts "Writing #{n} points" Benchmark.bm 12 do |bm| bm.report "write_points" do n.times do |i| write_point i, influxdb end end end
1 parent a309a32 commit 7a29338

File tree

1 file changed

+23
-6
lines changed

1 file changed

+23
-6
lines changed

lib/influxdb/client/http.rb

Lines changed: 23 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -43,13 +43,19 @@ def connect_with_retry
4343
delay = config.initial_delay
4444
retry_count = 0
4545

46-
http = build_http
46+
http = get_http
4747

4848
begin
49+
http.start unless http.started?
50+
4951
yield http
5052
rescue *InfluxDB::NON_RECOVERABLE_EXCEPTIONS => e
53+
http.finish
54+
5155
raise InfluxDB::ConnectionError, InfluxDB::NON_RECOVERABLE_MESSAGE
5256
rescue Timeout::Error, *InfluxDB::RECOVERABLE_EXCEPTIONS => e
57+
http.finish
58+
5359
retry_count += 1
5460
unless (config.retry == -1 || retry_count <= config.retry) && !stopped?
5561
raise InfluxDB::ConnectionError, "Tried #{retry_count - 1} times to reconnect but failed."
@@ -58,9 +64,8 @@ def connect_with_retry
5864
log(:warn) { "Failed to contact host #{host}: #{e.inspect} - retrying in #{delay}s." }
5965
sleep delay
6066
delay = [config.max_delay, delay * 2].min
67+
6168
retry
62-
ensure
63-
http.finish if http.started?
6469
end
6570
end
6671

@@ -130,11 +135,23 @@ def generate_cert_store
130135
store
131136
end
132137

138+
def get_http
139+
@https ||=
140+
begin
141+
https = config.hosts.map { |host|
142+
build_http host
143+
}
144+
145+
Hash[config.hosts.zip(https)]
146+
end
147+
148+
@https[config.next_host]
149+
end
150+
133151
# Builds an http instance, taking into account any configured
134152
# proxy configuration
135-
def build_http
136-
host = config.next_host
137-
port = config.port
153+
def build_http(host)
154+
port = config.port
138155

139156
http =
140157
if config.proxy_addr

0 commit comments

Comments
 (0)