Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Add json (serde) support #1081

Draft
wants to merge 17 commits into
base: master
Choose a base branch
from
Draft

WIP: Add json (serde) support #1081

wants to merge 17 commits into from

Conversation

cynecx
Copy link

@cynecx cynecx commented Jun 13, 2024

I started this work because I'm also working on adding Connect RPC support to tonic. So having proper protobuf json support is the first step into achieving that goal.

This (unfortunately very big) PR adds json/serde support to prost. This is still a WIP effort but it is almost feature complete and passes all conformance tests except tests related to fieldmasks.

A non-exhaustive list of TODOs:

  • Decide whether the whole "custom serde" design is the one we want to persue
  • Properly document code and public api
  • Properly put this all behind a feature flag
  • Probably split things up because this PR also includes a new Any implementation
  • ...

So I am just putting this up here to get a general vibe check...

Copy link
Collaborator

@caspermeijn caspermeijn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is impressive work and this feature is on the project roadmap (#624). However this PR is way too big to review. I did scroll through the code and ask some questions that came to mind.

Can you split this work into smaller PRs?

tests/src/lib.rs Outdated Show resolved Hide resolved
pub trait Sealed {}
}

pub trait AnyValue: CoreAny + Message + private::Sealed {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain what any v2 is and why it is better?

Copy link
Author

@cynecx cynecx Jun 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This basically tries to address the "improved Any" bulletpoint in #624. The new Any implementation provides support for JSON and "cross" serialization (JSON <-> Binary).

The summary for that change is basically going from:

struct OldAny {
    type_path: String,
    data: Vec<u8>,
}

to

struct NewAny {
    type_path: String,
    kind: enum {
        Protobuf(Vec<u8>),
        Json(JsonValue),
        Dyn(Box<dyn AnyValue>),
    }
}

prost-build/src/code_generator.rs Outdated Show resolved Hide resolved
@cynecx
Copy link
Author

cynecx commented Jun 14, 2024

However this PR is way too big to review.
Can you split this work into smaller PRs?

Yeah, it's actually on the TODO list. I just still haven't gotten to actually complete this as a whole yet.

@@ -172,6 +173,32 @@ impl Field {
_ => None,
}
}

pub fn json(&self) -> Option<Option<&Json>> {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't it make more sense to flatten this into Option<&Json>

Copy link
Author

@cynecx cynecx Jun 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Conceptionally yes, but this is an actual tri-state kind of thing, because (iirc) we are skipping code generation for Self::OneOf fields. But we could also make this more flat and just add another check for an oneof field at the callsite 🤷‍♂️.

@@ -107,7 +111,7 @@ impl fmt::Display for Duration {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
let mut d = *self;
d.normalize();
if self.seconds < 0 && self.nanos < 0 {
if self.seconds < 0 && self.nanos <= 0 || self.seconds <= 0 && self.nanos < 0 {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like an unrelated fix that can ship on its own.

@cblichmann
Copy link

Any news on this? This would be incredibly useful.

@cynecx
Copy link
Author

cynecx commented Nov 22, 2024

I've done a quick rebase and made few adjustments to pass the conformance tests again. It also seems like I need to do some tweaks to fix the CI for no-std.

Any news on this? This would be incredibly useful.

I am still way too busy with other things. What this PR needs is a split and a bit of slight documentation work. I will probably get to that at the end of December perhaps. Thanks for your patience!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants