Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Implements handling of partial frames by the streaming decompressor#88

Merged
SpringMT merged 1 commit intoSpringMT:v2SpringMT/zstd-ruby:v2from
gucki:real_streaming_descompressgucki/zstd-ruby:real_streaming_descompressCopy head branch name to clipboard
Jul 26, 2025
Merged

Implements handling of partial frames by the streaming decompressor#88
SpringMT merged 1 commit intoSpringMT:v2SpringMT/zstd-ruby:v2from
gucki:real_streaming_descompressgucki/zstd-ruby:real_streaming_descompressCopy head branch name to clipboard

Conversation

@gucki
Copy link
Contributor

@gucki gucki commented May 16, 2024

Fixes #87

Example use case:

read_buffer = "".force_encoding(Encoding::ASCII_8BIT)
loop do
  logger.debug("read partial")
  buf = reader.readpartial(64 * 1024)
  read_buffer << buf
  logger.debug("read #{buf.bytesize} bytes (#{read_buffer.bytesize} bytes buffered)")

  while !read_buffer.empty?
    result, data, read_buffer_pos = decompressor.decompress2(read_buffer)
    read_buffer.slice!(0, read_buffer_pos)
    logger.debug("decompressed #{data.bytesize} bytes (result #{result}, consumed #{read_buffer_pos} bytes, #{read_buffer.bytesize} left in read buffer)")
    writer.write(data)
    break if read_buffer_pos == 0
  end
end

It works according to the zstd streaming decompression documentation:

Use ZSTD_decompressStream() repetitively to consume your input.
The function will update both `pos` fields.
If `input.pos < input.size`, some input has not been consumed.
It's up to the caller to present again remaining data.
The function tries to flush all data decoded immediately, respecting output buffer size.
If `output.pos < output.size`, decoder has flushed everything it could.
But if `output.pos == output.size`, there might be some data left within internal buffers.,
In which case, call ZSTD_decompressStream() again to flush whatever remains in the buffer.
Note : with no additional input provided, amount of data flushed is necessarily <= ZSTD_BLOCKSIZE_MAX.
@return : 0 when a frame is completely decoded and fully flushed,
      or an error code, which can be tested using ZSTD_isError(),
      or any other value > 0, which means there is still some decoding or flushing to do to complete current frame :
                              the return value is a suggested next input size (just a hint for better latency)
                              that will never request more than the remaining frame size.

@SpringMT
Copy link
Owner

Thank you for the PR.
I'll check it later.

@SpringMT SpringMT changed the base branch from main to v2 July 26, 2025 13:57
@SpringMT SpringMT merged commit b402a77 into SpringMT:v2 Jul 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Streaming decompression questions

2 participants

Morty Proxy This is a proxified and sanitized view of the page, visit original site.