Replies: 1 comment 1 reply
-
Unfortunately, I don't know whether gzip has support to avoid the deadlock you mentioned, but using it on the message layer should work. |
Beta Was this translation helpful? Give feedback.
1 reply
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hey,
I'm running a setup in which I'm sending and receiving
Framed
messages via TCP. Specifically, it looks something like this:Now, this is using quite a lot of bandwidth. I can't swap out the JSON part for legacy reasons, so I thought about compression instead.
I see two approaches, each with their own drawbacks.
This is the first:
This is relatively straightforward. I get a bunch of bytes by JSON-encoding, I compress those bytes and pass them on to be framed and sent.
However, this is probably suboptimal wrt. compression, as the compressor state is reset for each message. (see edit below, I was confused)
A solution to fix that would be this construction:
This makes sense in my head because gzip is a streaming algorithm.
The
Framed
docs state that:Which makes me believe this is the proper way to do this. However, after searching for quite a while, I cannot find anything that provides a unified gzip reader and write. I could use the async compression crate and something like duplexify to... split the TCP stream, add gzip compression/decompression, merge it back together, and then add framing on top?
In addition to that, I'm worried about flushing the gzipped TCP stream. In particular, I'm wondering if something like this works or deadlocks:
EDIT: While re-reading my question and the docs, it seems the first approach is the intended one (i.e., layering gzip on top of framing). I think I was confused because Framed also wants a unified Stream/Sink, so I thought that was the one the docs were talking about.
Maybe the intended usage is to send gzipped chunks, one for each message, without resetting the encoder in between?
I am now utterly confused. Is there a recommended way of doing this?
Beta Was this translation helpful? Give feedback.
All reactions