FHIR JSON Serializer Problems with a custom resource

[ewoutkramer]

Discussed in #2579

^{Originally posted by Sneedd September 1, 2023}
Help! Been stuck a while now. Tried different approaches to make it work, but so far nothing worked.

Basically I am trying to serialize and deserialize an custom resource. Tried the System.Text.Json and Newtonsoft approach.

Here is my code, if you like to try it out: Create a new MSTest .NET6 project, add the Hl7.Fhir.R4 (v5.3.0) library and paste the following code. I added the results over the tests, where 5 of 8 are failing.

// Hl7.Fhir.R4 5.3.0
using Hl7.Fhir.Introspection;
using Hl7.Fhir.Model;
using Hl7.Fhir.Serialization;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using System;
using System.Collections.Generic;
using System.Runtime.Serialization;
using System.Text.Json;


namespace Example
{
    public class Workspace
    {
        public string Id { get; set; }
        public string? Name { get; set; }
        public bool? Active { get; set; }
    }

    [Serializable]
    [DataContract]
    [FhirType("FhirWorkspace", "http://hl7.org/fhir/StructureDefinition/FhirWorkspace", IsResource = true)]
    public class FhirWorkspace : DomainResource
    {
        private readonly Workspace _original;

        [IgnoreDataMember]
        public Workspace Original => _original;

        public override string TypeName => nameof(FhirWorkspace);

        public override IEnumerable<ElementValue> NamedChildren
        {
            get
            {
                foreach (var elementPair in base.NamedChildren)
                {
                    yield return elementPair;
                }
                if (Name != null)
                {
                    yield return new ElementValue("name", Name);
                }
                if (Active != null)
                {
                    yield return new ElementValue("active", Active);
                }
            }
        }

        [DataMember]
        [FhirElement("name", Order = 90)]
        public FhirString Name
        {
            get => new FhirString(_original.Name);
            set => _original.Name = value?.Value;
        }

        [DataMember]
        [FhirElement("active", Order = 100)]
        public FhirBoolean Active
        {
            get => new FhirBoolean(_original.Active);
            set => _original.Active = value?.Value;
        }

        public FhirWorkspace()
            : this(new Workspace())
        {
        }

        public FhirWorkspace(Workspace original)
        {
            _original = original;
        }

        public override IDeepCopyable DeepCopy()
        {
            return new FhirWorkspace(new Workspace { Name = _original.Name });
        }

        protected override IEnumerable<KeyValuePair<string, object>> GetElementPairs()
        {
            foreach (var elementPair in base.GetElementPairs())
            {
                yield return elementPair;
            }
            if (Name != null)
            {
                yield return new KeyValuePair<string, object>("name", Name);
            }
            if (Active != null)
            {
                yield return new KeyValuePair<string, object>("active", Active);
            }
        }
    }

    [TestClass]
    public class FhirSerialization
    {
        private void TestJson1SerializerWith(Workspace workspace)
        {
            var serializer = new FhirJsonSerializer();
            var jsonContent = serializer.SerializeToString(new FhirWorkspace(workspace));

            var parser = new FhirJsonParser();
            var fhirWorkspace = parser.Parse<FhirWorkspace>(jsonContent);
            Assert.IsNotNull(fhirWorkspace);
            var workspace2 = fhirWorkspace.Original;

            Assert.AreEqual(workspace.Name, workspace2.Name);
            Assert.AreEqual(workspace.Active, workspace2.Active);
        }

        // Fails: System.FormatException: While building a POCO: Element 'name' must not repeat (at FhirWorkspace.name[0])
        [TestMethod]
        public void SerializerJson1Test()
        {
            TestJson1SerializerWith(new Workspace { Name = "ABC", Active = true });
        }

        // Fails: System.FormatException: While building a POCO: Element 'active' must not repeat (at FhirWorkspace.active[0])
        [TestMethod]
        public void SerializerJson1_EmptyString_Test()
        {
            TestJson1SerializerWith(new Workspace { Name = "", Active = true });
        }

        // Fails: System.FormatException: While building a POCO: Element 'active' must not repeat (at FhirWorkspace.active[0])
        [TestMethod]
        public void SerializerJson1_NullString_Test()
        {
            TestJson1SerializerWith(new Workspace { Name = null, Active = true });
        }

        // Ok
        [TestMethod]
        public void SerializerJson1_AllNull_Test()
        {
            TestJson1SerializerWith(new Workspace { Name = null, Active = null });
        }

        private void TestJson2SerializerWith(Workspace workspace)
        {
            var modelInspector = new ModelInspector(Hl7.Fhir.Specification.FhirRelease.R4);
            modelInspector.ImportType(typeof(FhirWorkspace));

            var options = new JsonSerializerOptions()
                .ForFhir(modelInspector);

            var jsonContent = JsonSerializer.Serialize(new FhirWorkspace(workspace), typeof(FhirWorkspace), options);

            var fhirWorkspace = JsonSerializer.Deserialize(jsonContent, typeof(FhirWorkspace), options) as FhirWorkspace;
            Assert.IsNotNull(fhirWorkspace);
            var workspace2 = fhirWorkspace.Original;

            Assert.AreEqual(workspace.Name, workspace2.Name);
            Assert.AreEqual(workspace.Active, workspace2.Active);
        }

        // OK
        [TestMethod]
        public void SerializerJson2Test()
        {
            TestJson2SerializerWith(new Workspace { Name = "ABC", Active = true });
        }

        // Fails: Properties cannot be empty strings
        [TestMethod]
        public void SerializerJson2_EmptyString_Test()
        {
            TestJson2SerializerWith(new Workspace { Name = "", Active = true });
        }

        // OK
        [TestMethod]
        public void SerializerJson2_NullString_Test()
        {
            TestJson2SerializerWith(new Workspace { Name = null, Active = true });
        }

        // Fails: An object needs to have at least one property
        [TestMethod]
        public void SerializerJson2_AllNull_Test()
        {
            TestJson2SerializerWith(new Workspace { Name = null, Active = null });
        }
    }
}

So my problems are:

• Should not the two serializers behave the same?
• I do not understand the "Element 'xx' must not repeat" messages.
• Is it really not allowed to have an entity without any property value?
• Why empty strings are a problem?
• What I am doing wrong?

[mmsmits]

Refinement:

• Run unit test to check if something unexpected occurs.
• After that, maybe create issues based on these unit tests.
• Especially look at the "Element 'xx' must not repeat" messages. Is this a bug?

[Kasdejong]

Should not the two serializers behave the same?

The two (de)serializers are inherently different, and the "legacy" element model (serializer 1) will generally accept more (usually non-fhir-compliant) cases. We recommend using the newer ones (serializer 2) for more accurate error reporting

I do not understand the "Element 'xx' must not repeat" messages.

After some testing of your code I found that this is due to the Active/Name property of your FhirWorkspace class never being null. Basically, our SDK checks if a property "already" has a value, and if so, throws an error if that property is not a collection. To prevent this error from being thrown, simply initialize the FhirBoolean to null if the internal boolean was originally null. The same counts for the name (although .NET's nullability annotations are inconsistent between these types). A fix would be along the lines of:

[FhirElement("name", Order = 90)]
public FhirString? Name
{
    get => _original.Name is not null ? new FhirString(_original.Name) : null;
    set => _original.Name = value?.Value;
}

[DataMember]
[FhirElement("active", Order = 100)]
public FhirBoolean? Active
{
    get => _original.Active.HasValue ? new FhirBoolean(_original.Active) : null;
    set => _original.Active = value?.Value;
}

Is it really not allowed to have an entity without any property value?

This is technically allowed. Our implementation ignores the resource if it is empty and throws a DeserializationFailedException with issueSeverity "Warning". You can safely catch this exception.

Why empty strings are a problem?

Note that when present, elements cannot be empty - they SHALL have a value attribute, child elements, or extensions. The element's value attribute/property can never be empty. Either it is absent, or it is present with at least one character of non-whitespace content. (http://hl7.org/fhir/conformance-rules.html). This error is not fatal, though, so you could try to catch it. Just note that an empty string is equivalent to a string? with value null