Story/1045/cdm serialization

[davidalk]

Custom Serializer/Deserializer for Rune DSL Common Library

Overview

This change introduces a custom JSON serializer and deserializer for the Rune DSL Common Library. The primary goal is to ensure JSON serialization and deserialization that:

• Aligns with the Rune DSL Structure: The JSON output mirrors the semantic and structural design of the DSL for consistency and accuracy.
• Is Human-Readable: The generated JSON is intuitive and easy to interpret, enhancing usability and debugging.

The following specification has been followed when Implementing these changes Serialization Specification v0.9.pdf

Benefits to Users

• Seamless conversion between Rune DSL objects and JSON with minimal effort.
• Clear, human-readable JSON that reflects the natural hierarchy and relationships of the DSL.
• Test coverage ensures round trip of DSL objects between in-memory representations and serialized JSON.

High-Level Summary of the Development Approach

The RuneJJsonObjectMapper serves as the core component, offering pre-configured behaviour to ensure JSON output aligns closely with the structure of the DSL.

The implementation of the RuneJsonObjectMapper and RuneJsonObjectWriter provides a specialized solution for JSON serialization and deserialization tailored to the Rune DSL.

To enrich the JSON output with contextual information, the RuneJsonObjectWriter extends Jackson’s ObjectWriter. It adds top-level metadata, such as model name, type, and version, derived from annotations on the serialized objects.

Additionally, there are three annotations relied upon by the RuneJsonAnnotationIntrospector to customize the serialized structure. The RuneJsonAnnotationIntrospector is responsible for scanning the annotations on the generated Rune DSL POJOs and determining how they should be applied during both serialization and deserialization. It recognizes the following annotations:

• @RuneDataType: Captures key details about the object, such as its canonical type name, model name and version. This information is then injected into the top-level JSON metadata during serialization.
• @RuneAttribute: Marks specific fields within the object as attributes, guiding how they are serialized to ensure they align with the DSL's structure.
• @RuneMetaType: Indicates when nodes with an extra wrapping layer (originating from the underlying Java POJO) should be unwrapped during serialization, ensuring that the resulting JSON structure is clean and free of unnecessary nesting.

These annotations collectively ensure that the serialized JSON output reflects both the structure and semantics of the Rune DSL. See finos/rune-dsl#889 for annotation implementation details.

Findings

• Property ordering can't be overridden when using JSON unwrapping functionality, see jackson-databind issue 1670 for further details.

• Top level meta types are not straight forward due to not being aware of when you are processing a top level type in Jackson. This solution achieves this by overriding the ObjectMapper and ObjectWriter

• The specification for @key:scoped requires that we go from an array structure in the old world to a single element in the new world:

  Existing

  "quantity": [
    {
      "value": {
        "value": 150000,
        "unit": {
          "financialUnit": "Share"
        }
      },
      "meta": {
        "location": [
          {
            "scope": "DOCUMENT",
            "value": "quantity-9"
          }
        ]
      }
    }
  ]

  New

  "quantity": [
    {
      "@key:scoped": "quantity-9",
      "value": 150000,
      "unit": {
        "financialUnit": "Share"
      }
    }
  ]

  This has been achieved by adding new getters/setters to com.rosetta.model.metafields.MetaFields in the DSL. These new methods are only annotated with the new @RuneAttribute annotations and add a single item to the underlying data structure. See Rune DSL PR 899

• The dynamically generated and compiled test .rosetta files is done inside the RuneSerializerTestHelper.generateCompileAndGetRootDataType() method using an already existing CodeGeneratorTestHelper utility. There is a problem in that the generated classes are not visible to the Jackson mapper used in the test. Upon investigation it looked like this is a class loader issue that I was not able to get to the bottom of quickly. As a work around I've stored the generated classes inside a test class called DynamicCompiledClassLoader which is handed of to the mapper. It may be worth investigating this further in case the issue is not only isolated to the tests.

• Although not relevant for this change this work has highlighted a problem in the current RosettaObjectMapper regarding setting multi cardinality values. The mapper is currently relying on a legacy fallback LegacyRosettaBuilderIntrospector in order to be able to set multi cardinality values correctly. On investigation it looks like the reason for this is due to the @RosettaAttribute annotation having been moved from set(List<Foo> ...) methods to add(Foo ...) methods which Jackson is not able to work with natively. This change adds the new @RuneAttribute annotation back to the setter methods which means the newRuneJacksonJsonObjectMapper requires no legacy fallback.

Follow Up Tasks

• Enable currently disabled error handling tests and implement error handling logic.
• Investigate how we can enforce property ordering.

[hugohills-regnosys]

DSL version