FlattenedString Serializer Can not convert a string from third app properly

[russellgsystems]

I am receiving a payload from a third party software. The payload is a JSON string. I am using the Decode Application Message.vi with a string as the constant as the data type for the variant. But when I get error 116 (Flattened String To Variant in OpenSerializer.lvlib:Serializer.FlattenedString.lvclass:unmarshalling.vi:3430001->OpenSerializer.lvlib:Serializer.lvclass:UnMarshall.vi->OpenSerializer.lvlib:Serializer.lvclass:UnMarshall (binary).vi->MQTT Base.lvlib:MQTT_Base.lvclass:Decode Publish Package.vi:2830001->MQTT Client.lvlib:Client.lvclass:Decode Application Message (Type).vi->Test MQTT Subscribe.vi).

If I dig inside the VI and add a byte array to string VI then the JSON string looks correct see screenshot below. So that is my work around for now. I saw that you had a plain text to variant VI and tried that but it also did not work. I am wonder if their some other way to do this instead of just taking the byte array and converting it to a string.

[francois-normandin]

@russellgsystems
You need to set the serializer to Serializer.JSON.
The default serializer only accepts the LabVIEW Flatten/Unflatten method. For anything else (Base64, XML, JSON or your own implementation), you need a more specific class.

In this case, install OpenSerializer.JKI-JSON with VIPM.

Look for the OpenSerializer.JKI-JSON palette and use the top left icon (drop VI Content) to specifiy a serializer of type "JSON".

Navigate to the "MQTT Base" palette (next to MQTT Client) and drop the "Set Serializer" method right after the MQTT Client Create node. This will set the Serializer class that will decode your raw packet into the correct format.

Alternatively, if you can receive more than one type of flattened data (JSON, XML, Raw, etc.) on the same connection, you can use the "Decode Application Message (Raw)" method from the Client palette to extract the payload directly, and choose your own deserializer, specific to the topic received.

[francois-normandin]

As a bonus, here is the video that details the OpenSerializer and how to extend the base class to support any custom flattening schemes.

https://www.youtube.com/watch?v=Y_GyZVHIORI

[russellgsystems]

Awesome thank you. I would like to use the JKI JSON but I am afraid it might not work because the structure of my JSON can change for the same topic. Do you know if the JSON structure must always be the same for the JKI JSON to work or if it can change?

[francois-normandin]

I selected JKI JSON instead of jsonText because if you leave the Variant input empty, the JKI JSON will reconstruct the data into a cluster based on the type detected in the json object.

If this is not satisfactory and you would like to get the JSON object as a string, I suggest you use the raw payload and use the toString node as you did to retrieve the payload. Then you can use the JSON parser of your choice.

I personally prefer jsonText, but the JKI JSON is the only one that matches unknown types to a variant using a "best guess" first pass. There is also a "Adapt to Type" method if you know the real schema later on.

For instance:

[BarashkinRoman]

Dear Francois,
i have similar problem.

I'm receiving a payload from a third party software (Thingsboard), the payload is a string and receiving the raw data (probe 4 on the fig).

But if I'm using the Decode Application Message (Type).vi with a string as the constant as the data type for the variant I get

Error 116 occurred at Flattened String To Variant in OpenSerializer.lvlib:Serializer.FlattenedString.lvclass:unmarshalling.vi:3430001->OpenSerializer.lvlib:Serializer.lvclass:UnMarshall.vi:4390003->OpenSerializer.lvlib:Serializer.lvclass:UnMarshall (binary).vi:6550002->MQTT Base.lvlib:MQTT_Base.lvclass:Decode Publish Package.vi:2830001->MQTT Client.lvlib:Client.lvclass:Decode Application Message (Type).vi->DAQ.vi
Possible reason(s): LabVIEW: Unflatten or byte stream read operation failed due to corrupt, unexpected, or truncated data.

[francois-normandin]

Hi @BarashkinRoman ,

yes, this is expected behavior when using "Decode Application Message (Type).vi". This is because it is the mirror of the "Publish.vi" method which uses the default serializer (Flatten to String) which is a LabVIEW internal format. A string will be represented as a size-prepended string (4 bytes for length) followed by the raw bytes representing the characters in the string.
It is useful if you want to "Publish Anything", such as a complex cluster, without managing the serialization (Flatten to string) and deserialization (Unflattening from String) part yourself. However, you must have the exact same type on both sides (Publisher and Subscriber).

Try not to mix these two pairs of nodes:

When you are not publishing the data from LabVIEW and receiving it also in LabVIEW, you'd better use the raw payloads, unless you provide a different serializer for parsing the bytes.

In your particular case, Thingsboard is publishing the string in JSON format. To decode, you can do either of these things:

1- Use the "Decode Application Message (Raw).vi" and insert the "Byte Array to String" method. You can then parse your JSON string to extract the data you want.

2- Tell the client to use a JSON Serializer to interpret the incoming payload. To do that, you will need to set the serializer to "JSON". Make sure the JKI-JSON OpenSerializer package is installed and you will find the nodes in the MQTT Base (Set Serializer) and in the JSON Serializer palette for the type.

Ultimately, your app should set the serializer type before connecting to the server:

And then you can use either methods to access your data:

Method 1: Raw string (NO NEED TO SET SERIALIZER UPSTREAM)
Method 2: JSON Serialization (REQUIRES THAT YOU SET THE CLIENT SERIALIZER UPSTREAM)

All these paths will allow you to get to your data:

[francois-normandin]

Oh, and another possibility... especially if the data you receive is not always serialized with the same format (ex: one message is JSON formatted, and another is not...), then you can use the OpenSerializer methods directly on the raw bytes as below (swapping serializer type for each topic as needed):

[BarashkinRoman]

Thank you for the more than detailed and detailed answer!