[controller] Allow AdminOperationSerializer to use lower version to serialize the AdminOperation message #1549
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[controller] Allow AdminOperationSerializer to use lower version to serialize the AdminOperation message
adminOperationProtocolVersion
at AdminTopicMetadata on ZK (could be lower version) as the SOT for serializer to serialize the message and send that writer schema version along with the bytes to deserialize back at consumption stage.Serialization issues when serializing the object by lower version
This issue is figured during the effort of serialize the object with different schema that it was created with.
The use case that I used:
=> Even though v84 and v74 is compatible, the serialization failed due to ClassCastException (cannot cast Boolean to Number).
Debug process:
=> Then the field and position of the writer schema (v74) is basically not matched anymore (expecting pos 21 to be
replicationMetadataVersionID
(int), but gotwriteComputationEnabled
(boolean) from v84, causing cast exceptionjava.lang.Boolean cannot be cast to java.lang.Number
There are two options:
Option 1: Override
getField
to get the field by fieldNameOption 2: Turn the current AdminOperation (with latest version schema) to GenericRecord (with writer schema id) and serialize the GenericRecord with writer schema id.
Option 1 was implemented (PR) but then closed after getting the suggestion for option 2.
Option 2, though very "bulky", it works and cut the future maintenance cost since we were modifying Avro API. We chose this option over option 1 and the implementation can be found specifically here.
How was this PR tested?
UTs and integration tests
Does this PR introduce any user-facing changes?