Workaround for providing support for types JObject with System.Text.Json

[snjosephms]

We are migrating our code base from Newtonsoft to System.Text.Json. While testing this migration effort, we have encountered errors while working with Newtonsoft.Json.Linq.JObject objects

Issue during serialization:

var c = new DerivedClassA
            {
                F = JObject.FromObject(new
                {
                    Box = "abc",
                    Side = "def"
                })
            };

We observed that while serializing the data using System.Text.Json, the key is present in the serialized data, but the values are not.
"Properties":{"Box":[],"Side":[]}}]}

Issue during de-serialization:
We get the following error on deserialization:
"System.NotSupportedException: The collection type 'Newtonsoft.Json.Linq.JToken' is abstract, an interface, or is read only, and could not be instantiated and populated. Path: $.Message.Attachments[0].Properties.Box | LineNumber: 0 | BytePositionInLine: 934. ---> System.NotSupportedException: The collection type 'Newtonsoft.Json.Linq.JToken' is abstract, an interface, or is read only, and could not be instantiated and populated.   Stack Trace: 
ThrowHelper.ThrowNotSupportedException(ReadStack& state, Utf8JsonReader& reader, NotSupportedException ex)
IEnumerableOfTConverter2.CreateCollection(Utf8JsonReader& reader, ReadStack& state, JsonSerializerOptions options) JsonCollectionConverter2.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, TCollection& value)
JsonConverter1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value) JsonDictionaryConverter3.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, TDictionary& value)
JsonConverter1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value) JsonPropertyInfo1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
ObjectDefaultConverter1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value) JsonConverter1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
JsonCollectionConverter2.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, TCollection& value) JsonConverter1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
JsonPropertyInfo1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader) ObjectDefaultConverter1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
JsonConverter1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value) JsonPropertyInfo1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
ObjectWithParameterizedConstructorConverter1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value) JsonConverter1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
JsonConverter1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state) JsonSerializer.ReadFromSpan[TValue](ReadOnlySpan1 utf8Json, JsonTypeInfo jsonTypeInfo, Nullable1 actualByteCount) JsonSerializer.ReadFromSpan[TValue](ReadOnlySpan1 json, JsonTypeInfo jsonTypeInfo)
JsonSerializer.Deserialize[TValue](String json, JsonSerializerOptions options)"

Can you please share a workaround for providing support for JObject type with System.Text.Json?

[gregsdennis]

The analog for JToken is JsonNode. You'll need to change your models to this as the Newtonsoft models don't have STJ converters.

There's also JsonObject for JObject, and JsonArray for JArray. Other JSON value deserialize to JsonValue.

[eiriktsarpalis]

Here's an example implementation of a JsonConverter that can be used to serialize and deserialize JToken objects using the System.Text.Json library:

public class JTokenConverter : JsonConverter<JToken>
{
    public override bool CanConvert(Type typeToConvert) => typeof(JToken).IsAssignableFrom(typeToConvert);
    public override JToken Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
    {
        // Use JsonDocument to parse the JSON and create a JToken from it
        using JsonDocument document = JsonDocument.ParseValue(ref reader);
        return JToken.Parse(document.RootElement.GetRawText());
    }

    public override void Write(Utf8JsonWriter writer, JToken value, JsonSerializerOptions options)
    {
        // Write the raw JSON from the JToken to the writer
        writer.WriteRawValue(value.ToString());
    }
}

[gregsdennis]

Whole that may help in the immediate term, if they're migrating away from Newtonsoft, I think it makes sense to encourage a transition to JsonNode in the models.

The upside of having this converter is that they can continue to work through the transition instead of having to change everywhere all at once.

[benbenwilde]

value.ToString() makes indented json by default, so something like this is better:
writer.WriteRawValue(value.ToString(options.WriteIndented ? Newtonsoft.Json.Formatting.Indented : Newtonsoft.Json.Formatting.None));

[johnthcall]

My team has very large JObject paylods and so I looked into an approach without allocating a large string. In the below benchmark of serializing a JObject with the writer snippet the length of "small" JObjects string is 976 characters and the length of the "large" JObjects string is 256959 characters.

using (var memoryStream = (RecyclableMemoryStream)StreamManager.GetStream())
using (var streamWriter = new StreamWriter(memoryStream))
using (var textWriter = new Newtonsoft.Json.JsonTextWriter(streamWriter)
{
     Formatting = options.WriteIndented ? Newtonsoft.Json.Formatting.Indented : Newtonsoft.Json.Formatting.None
 })
{
    value.WriteTo(textWriter);
    textWriter.Flush();
    writer.WriteRawValue(memoryStream.GetReadOnlySequence());
}

Method	UseLarge	Mean	Allocated
stringJObject	False	3.916 us	7.04 KB
streamJObject	False	4.620 us	7.8 KB
streamJObject	True	315.877 us	508.58 KB
stringJObject	True	413.405 us	1522.92 KB

[aalstes]

What worked for me was converting the JObject to a dictionary with this code:

    public static Dictionary<string, object> ToDictionary(JObject jObject)
    {
        var dictionary = new Dictionary<string, object>();

        foreach (var property in jObject)
        {
            dictionary[property.Key] = ToObject(property.Value);
        }

        return dictionary;
    }

    public static object ToObject(JToken token)
    {
        if (token.Type == JTokenType.Object)
        {
            return ToDictionary((JObject)token);
        }
        else if (token.Type == JTokenType.Array)
        {
            return ((JArray)token).ToObject<object[]>();
        }
        else
        {
            return ((JValue)token).Value;
        }
    }

[Seabizkit]

i have a simlar... migration project which should be kicked off...

but with that i see it as, the refernace to newtonsoft have to be remove.... which requires both the producer and the consumer to stop using newtonsoft object.

im wondering, even if you get system.text.json to take a sting and then convert to a newton type, you still using a newton type and not removing the depenacy..

can someone help me understand the conversion plan here.. like you can create JsonConverter, but how does that help, im clearly missing something. @snjosephms

[johnthcall]

In some projects there may be classes from external nugets that have a JObject property where you do not control the move to JsonNode or JsonDocument. In other cases you do a slow move of service by service where the same contract may be serialized by either STJ or Newtonsoft. Eventually it will likely make sense to fully move to STJ but because of the designed differences parsing to JObject for quick lookups may be your hot path that you optimize for.