Struct AsyncStreamCDC

pub struct AsyncStreamCDC<R> { /* private fields */ }

An async-streamable version of the FastCDC chunker implementation from 2020 with streaming support.

Use new to construct an instance, and then as_stream to produce an async Stream of the chunks.

Both futures and tokio-based AsyncRead inputs are supported via feature flags. But, if necessary you can also use the async_compat crate to adapt your inputs as circumstances may require.

Note that this struct allocates a Vec<u8> of max_size bytes to act as a buffer when reading from the source and finding chunk boundaries.

async fn run() {
    let source = std::fs::read("test/fixtures/SekienAkashita.jpg").unwrap();
    let mut chunker = AsyncStreamCDC::new(source.as_ref(), 4096, 16384, 65535);
    let stream = chunker.as_stream();

    let chunks = stream.collect::<Vec<_>>().await;

    for result in chunks {
        let chunk = result.unwrap();
        println!("offset={} length={}", chunk.offset, chunk.length);
    }
}

Implementations

impl<R: AsyncRead + Unpin> AsyncStreamCDC<R>

pub fn new(source: R, min_size: u32, avg_size: u32, max_size: u32) -> Self

Construct a StreamCDC that will process bytes from the given source.

Uses chunk size normalization level 1 by default.

pub fn with_level( source: R, min_size: u32, avg_size: u32, max_size: u32, level: Normalization, ) -> Self

Create a new StreamCDC with the given normalization level.

pub fn as_stream(&mut self) -> impl Stream<Item = Result<ChunkData, Error>> + '_