flate2::Decompress::decompress consumes more input than actually used

[link2xt]

flate2::Decompress::decompress is documented as "consuming only as much input as needed and writing as much output as possible". However in my tests it consumes the whole input even if output buffer size is not sufficient. Because of this I cannot use flate2 to decode the IMAP stream incrementally using async-compression crate: chatmail/async-imap#112

Here is a failing test where output buffer is only 8 bytes long, but the whole input is consumed:

#[test]
fn deflate_decoder_partial() {
    // Decompresses to
    // "* QUOTAROOT INBOX \"User quota\"\r\n* QUOTA \"User quota\" (STORAGE 76 307200)\r\nA0001 OK Getquotaroot completed (0.001 + 0.000 secs).\r\n"
    let input = vec![
        210, 82, 8, 12, 245, 15, 113, 12, 242, 247, 15, 81, 240, 244, 115, 242, 143, 80, 80, 10,
        45, 78, 45, 82, 40, 44, 205, 47, 73, 84, 226, 229, 210, 130, 200, 163, 136, 42, 104, 4,
        135, 248, 7, 57, 186, 187, 42, 152, 155, 41, 24, 27, 152, 27, 25, 24, 104, 242, 114, 57,
        26, 24, 24, 24, 42, 248, 123, 43, 184, 167, 150, 128, 213, 21, 229, 231, 151, 40, 36, 231,
        231, 22, 228, 164, 150, 164, 166, 40, 104, 24, 232, 129, 20, 104, 43, 128, 104, 3, 133,
        226, 212, 228, 98, 77, 61, 94, 46, 0, 0, 0, 0, 255, 255,
    ];

    // Create very small output buffer.
    let mut output = vec![0; 8];

    let zlib_header = false;
    let mut decompress = flate2::Decompress::new(zlib_header);

    let flush_decompress = flate2::FlushDecompress::None;
    let status = decompress.decompress(&input, &mut output, flush_decompress).unwrap();
    assert_eq!(status, flate2::Status::Ok);

    // Should not consume everything, there is not enough space in the buffer for the output.
    assert_ne!(decompress.total_in(), input.len() as u64);
}

Here miniz_oxide::inflate::stream::inflate returns bytes_consumed equal to the whole input size even though there was not enough space in the output buffer:

flate2-rs/src/ffi/rust.rs

Line 73 in 1a28821

self.total_in += res.bytes_consumed as u64;

I thought about reporting the bug directly to miniz_oxide, but don't see in its documentation any guarantee that it will consume only as much as needed.

[link2xt]

The test passes with --no-default-features --features zlib, so probably should be upstreamed to miniz_oxide.

[link2xt]

I upstreamed the issue to miniz_oxide: Frommi/miniz_oxide#158

[Byron]

Decompress is a low-level building block which is much trickier to use than the high-level IO wrappers.
As stated in decompress() multiple calls may be required to finish, and receive a status that is StreamEnd.

My recommendation is to check how Decompress used in the flate2 codebase, or take a look at its tests.

[Byron]

So it looks like a bug with minz_oxide, right? Or does flate2 have implementation specific to miniz_oxide?
If so it's the latter, a test and fix would definitely be welcome.

[link2xt]

Ah, I see. Would that then not be an issue with miniz_oxide that would have to be fixed there?

I upstreamed the issue to Frommi/miniz_oxide#158, see my second message in this issue.

The bug should be fixed in miniz_oxide, or flate2 should use low-level API https://docs.rs/miniz_oxide/latest/miniz_oxide/inflate/core/fn.decompress.html which has no such bug.

[link2xt]

I am not sure why this is converted from an issue to a discussion, even if underlying library is buggy it is still a bug in flate2 that should be fixed either by upgrading to fixed miniz_oxide or switching to API that has no such bug. Currently documentation for flate2 says it does not consume more than needed, but with the default implementation it is not true.

[Byron]

Can you recreate a PR that contains the sample above as test-case? Then it would fail with the default backend, and should succeed with the others.
From there, it should be possible to figure out which patch to miniz_oxide solves the problem so a patch can be posted there.

[link2xt]

I created #436, but had to reorder the tests as all zlib and miniz_oxide tests are in the same job and the job fails on the first failing step.