Add zstd file compression support #93
Conversation
|
Thanks for the PR. So this compresses each file individually and still uses the iroh-blobs protocol as is. For lots of small files you would get better results by compressing the entire transfer stream, but that would have to be a different ALPN and would require changes in blobs itself I suppose. Zstd is also a very good choice. I love both zstd and lz4 from Yann Collet. One thing that is a bit unfortunate is that zstd adds a C dependency. LZ4 would obviously have not nearly as good compression. I will take a look some time this week. |
|
I was also considering the case of sending many small files. One option is to tar the directory before sending, but that approach doesn’t allow for parallelization. Ideally, we’d handle it the way you described, though that would require changes to the blob's logic. |
|
An interesting variant would be to have the entire interaction under a different ALPN, e.g. "/iroh-bytes/4-zstd". Then write some logic that just handles the wrapping compression/decompression. But that would be a bit of work, since our ProtocolHandler and friends does not allow you to abstract over the stream type. It is definitely an interesting challenge to make this simpler inside iroh. |
|
FYI I don't have forgotten about this. I think the best way to get compression is to compress the entire data stream on the wire, not compress individual blobs. The issue is that until now it was not possible to abstract over the stream type that is used by the provide and get side of blobs, so compressing individual blobs is the only thing you can implement. In a blobs release soon, there will be a way to abstract over the stream type. See n0-computer/iroh-blobs#147 and the compression example in there. |
|
Tracked in #106 |
## Description
This PR introduces abstract versions of iroh::endpoint::{SendStream,
RecvStream} and modifies the provide side and get side implementation to
use these abstract streams instead of directly using iroh::endpoint
streams.
This is necessary for wrapping the streams into a transformation such as
compression, see the discussion in
n0-computer/sendme#93 .
The compression example shows how streams can be wrapped into
compression/decompression to create a derived protocol with a different
ALPN that is identical to the blobs protocol except for compression.
## Breaking Changes
iroh::endpoint::SendStream and iroh::endpoint::RecvStream are replaced
with the traits iroh_blobs::util::SendStream and
iroh_blobs::util::RecvStream in the get FSM and in the provider side
API.
## Notes & open questions
<!-- Any notes, remarks or open questions you have to make about the PR.
-->
## Change checklist
- [ ] Self-review.
- [ ] Documentation updates following the [style
guide](https://rust-lang.github.io/rfcs/1574-more-api-documentation-conventions.html#appendix-a-full-conventions-text),
if relevant.
- [ ] Tests if relevant.
- [ ] All breaking changes documented.
---------
Co-authored-by: Frando <frando@unbiskant.org>
2 participants
This pull request adds support for compressing files before sending them using Zstandard. Compression is enabled with the
-zflag, and the compression quality can be configured via the-qflag.The feature is gated behind the zstd feature flag. Removing this feature disables compression support entirely.
This change introduces a new dependency on the
async-compressioncrate.If a user attempts to send compressed files to a build that does not support zstd decompression, the system will add a .zst extension to each file and display the following warning: