What could be the reason for read() on a File to not utilize the full buffer size?

[shameempk]

I have an instance of File from tokio::fs::file. When I do read().await on that file, it doesn't read to the full capacity of the buffer. read() always reads a fixed bytes of data.

Snippet:

let mut buffer = [0; 64 * 1024];
while let Ok(count) = self.source.read(&mut buffer[..]).await {
    if count > 0 {
        // count here is always 16KB. Note that the buffer can accommodate 64KB
        yield(ret)
    } else {
        return;
    }
}

Why is the count always 16KB instead of 64KB?

[shameempk]

One of my colleague pointed me to https://github.com/tokio-rs/tokio/blob/master/tokio/src/io/blocking.rs#L29. But what is the reasoning behind this?

[Darksonn]

We had a discussion about increasing this limit previously, but we never got around to doing so.

[shameempk]

What are my options if I want to read buffer.len() bytes in a non-blocking fashion?

[Darksonn]

You could use spawn_blocking directly to make the blocking read yourself. (This is how tokio::fs::File works internally.) The other option is to submit a PR to increase the limit.

[shameempk]

I ended up using async_std::fs at the end.