v1.5.6.4

What's Changed

• fix: free dctx for Zstd.decompress by @SpringMT in #81
• fix: compression crash by @SpringMT in #83

Bug Fix

• Fixed a memory leak when using Zstd.decompress().
  In versions 1.5.6.2 and 1.5.6.3, the context (ZSTD_DCtx) does not released after the decompression process was completed. This PR #81 addresses that issue.

• Resolved a crash when using Zstd.compress().
  In versions 1.5.6.2 and 1.5.6.3, Zstd.compress() internally used ZSTD_compressStream2, but it was crashing due to incomplete consumption of the internal buffer when large bytes were received. Zstd.compress() now uses ZSTD_compress2 instead of ZSTD_compressStream2 to ensure it doesn’t crash when processing large bytes.

If you are using versions v1.5.6.2 or v1.5.6.3, please update to version v1.5.6.4.

Full Changelog: v1.5.6.3...v1.5.6.4

v1.5.6.3

What's Changed

• Feature/unlock gvl for streaming compression/decompression by @SpringMT in #79

Full Changelog: v1.5.6.2...v1.5.6.3

Improvemnt

Unlock GVL for streaming compression/decompression

Zstd streaming compression/decompression is a CPU-only process that do not involve the ruby interpreter.
Therefore, it is possible to unlock the GVL, allowing for parallel multiple threads, thus fully utilizing CPU resources.

This program demonstrates the difference:
(in benckmarks)

Re-introduce #53

Streaming Compression

$LOAD_PATH.unshift '../lib'
require 'zstd-ruby'
require 'thread'

GUESSES = (ENV['GUESSES'] || 1000).to_i
THREADS = (ENV['THREADS'] || 1).to_i

p GUESSES: GUESSES, THREADS: THREADS

sample_file_name = ARGV[0]
json_string = File.read("./samples/#{sample_file_name}")

queue = Queue.new
GUESSES.times { queue << json_string }
THREADS.times { queue << nil }
THREADS.times.map {
  Thread.new {
    while str = queue.pop
      stream = Zstd::StreamingCompress.new
      stream << str
      res = stream.flush
      stream << str
      res << stream.finish
    end
  }
}.each(&:join)

Without this patch:

[springmt@MacBook-Pro] (main)✗ % time THREADS=4 bundle exec ruby multi_thread_streaming_comporess.rb city.json
{:GUESSES=>1000, :THREADS=>4}
THREADS=4 bundle exec ruby multi_thread_streaming_comporess.rb city.json  2.83s user 0.29s system 94% cpu 3.299 total

With the patch:

[springmt@MacBook-Pro] (feature/unlock-gvl)✗ % time THREADS=4 bundle exec ruby multi_thread_streaming_comporess.rb city.json
{:GUESSES=>1000, :THREADS=>4}
THREADS=4 bundle exec ruby multi_thread_streaming_comporess.rb city.json  3.33s user 0.36s system 266% cpu 1.385 total

Streaming Decompression

$LOAD_PATH.unshift '../lib'
require 'zstd-ruby'
require 'thread'

GUESSES = (ENV['GUESSES'] || 1000).to_i
THREADS = (ENV['THREADS'] || 1).to_i

p GUESSES: GUESSES, THREADS: THREADS

sample_file_name = ARGV[0]
json_string = File.read("./samples/#{sample_file_name}")
target = Zstd.compress(json_string)

queue = Queue.new
GUESSES.times { queue << target }
THREADS.times { queue << nil }
THREADS.times.map {
  Thread.new {
    while str = queue.pop
      stream = Zstd::StreamingDecompress.new
      stream.decompress(str)
      stream.decompress(str)
    end
  }
}.each(&:join)

Without this patch:

[springmt@MacBook-Pro] (main)✗ % time THREADS=4 bundle exec ruby multi_thread_streaming_decomporess.rb city.json
{:GUESSES=>1000, :THREADS=>4}
THREADS=4 bundle exec ruby multi_thread_streaming_decomporess.rb city.json  2.03s user 0.28s system 93% cpu 2.486 total

With the patch:

[springmt@MacBook-Pro] (feature/unlock-gvl)✗ % time THREADS=4 bundle exec ruby multi_thread_streaming_decomporess.rb city.json
{:GUESSES=>1000, :THREADS=>4}
THREADS=4 bundle exec ruby multi_thread_streaming_decomporess.rb city.json  2.49s user 0.49s system 227% cpu 1.310 total

v1.5.6.2

What's Changed

• feat: add StreamWriter and StreamReader by @SpringMT in #76
• Add support for dictionary use with streaming operations by @AndrewCEmil in #62
• Feature/dictionary standardized by @SpringMT in #77

Feature

• Zstd.compress supports :level and :dict keyword args
• Zstd.decompress supports :dict keyward args
• Zstd::StreamingCompress.new supports :level and :dict keyword args
• Zstd::StreamingDecompress.new supports :dict keyword args

Improvemnt

• Zstd.compress uses ZSTD_compressStream2 instead of ZSTD_compress
• Zstd.decompress uses ZSTD_decompressDCtx instead of ZSTD_decompress

Both of these changes maintain compatibility with previous versions.

Deprecation

• Zstd.compress_using_dict adds deprecation warning
  • use Zstd.compress with :dict keyword args
• Zstd.decompress_using_dict adds deprecation warning
  • use Zstd.decompress with :dict keyword args
• Zstd.compress with level args add deprecation warning
  • use Zstd.compress with :level keyword args
• Zstd::StreamingCompress.new with level args add deprecation warning
  • use Zstd::StreamingCompress.new with :level keyword args

New Contributors

• @AndrewCEmil made their first contribution in #62

Full Changelog: v1.5.6.1...v1.5.6.2

v1.5.6.1

What's Changed

• chore: Configure Renovate by @renovate in #68
• feat: update bechmarks by @SpringMT in #72
• docs: update benchmark result by @SpringMT in #73
• fix: incompatible function pointer types at streaming_decompress_compact and streaming_compress_compact by @SpringMT in #75
  • Related with #74

New Contributors

• @renovate made their first contribution in #68

Full Changelog: v1.5.6.0...v1.5.6.1

v1.5.6.0

What's Changed

• feat: add ruby 3.3 for GitHub Actions by @SpringMT in #66
  • Related with #63
• feat: update Zstd 1.5.6 by @SpringMT in #67

Full Changelog: v1.5.5.1...v1.5.6.0

v1.5.5.1

What's Changed

• Fix Readme Typo by @PatoSoft in #60
• Support GC compaction by @Watson1978 in #58
• feat: ignore .vscode by @SpringMT in #65

New Contributors

• @PatoSoft made their first contribution in #60

Full Changelog: v1.5.5.0...v1.5.5.1

v1.5.5.0

Update zstd v1.5.5

What's Changed

• Update zstd to v1.5.5 by @SpringMT in #57

Full Changelog: v1.5.4.1...v1.5.5.0

Release v1.5.4.1

Support skippable flame.

Release v1.5.4.0

Update zstd v1.5.4

Release v1.5.2.3

Support compress and decompress using dictionary.
See https://github.com/SpringMT/zstd-ruby#compression-using-dictionary and https://github.com/SpringMT/zstd-ruby#decomporession-using-dictionary.

Breaking changes

Decompression frame check is strictly

Previous version use ZSTD_getDecompressedSize when check decompression string, but this method is obsolete.
So, v1.5.2.3 zstd-ruby gem uses ZSTD_getFrameContentSize .
https://raw.githack.com/facebook/zstd/release/doc/zstd_manual.html#Chapter3

In ZSTD_getFrameContentSize, an empty string with decompression is invalid, so in the spec, an empty string decompression should have been treated as an error.