allow type casting in dynamic schema

[aleimu]

#639

When the Schema Type is Dynamic, after JSON serialization, the float loses precision and becomes an int type value. When storing such a value back into the dynamic database, it causes a type error. How can this issue be resolved?

  {"id": 111,"action": "test","user_id": 0,"soc": 1.01,"mileage": 123.02}
  {"id": 111,"action": "test","user_id": 0,"soc": 1.0,"mileage": 123.0}

POST  /api/v1/ingest  HTTP/1.1
HOST   : 192.168.33.10:8002
HEADERS:
        Authorization: Basic 111111=
        Content-Type: application/json
        User-Agent: go-resty/2.15.3 (https://github.com/go-resty/resty)
        X-P-Stream: test
BODY   :
[
   {
      "id": 111,
      "action": "test",
      "user_id": 0,
      "soc": 1,
      "mileage": 123
   }
]

------------------------------------------------------------------------------
~~~ RESPONSE ~~~
STATUS       : 400 Bad Request
PROTO        : HTTP/1.1
....................
BODY         :
Invalid Request: Could not process this event due to mismatch in datatype
==============================================================================
sock file [/var/qlog_platform.sock] not found, uds log ignored

[nitisht]

Thanks for reporting @aleimu . Can you try the new custom schema mechanism detailed here: https://www.parseable.com/blog/parseable-v1.6.0-release#generate-schema-for-a-log-event

With this you get to specify the schema yourself before sending events so you can take the best call and specify the correct type for a field.

This should solve the issue - feel free to let us know if there are other concerns.

[aleimu]

Thanks for answer,I've tried this way,but once it's set as a static schema
, the possibility of change is lost. There might be additional fields to be added in the future. In my consideration, a dynamic should be able to make the float type support all numeric types including int. However, it's still not supported in the 1.6.0 version of the dynamic schema that I'm currently using. Let's make the dynamic schema even more dynamic.

# create dynamic schema by curl
curl -H 'X-P-Stream: test_dynamic' -H 'Authorization: Basic YWRtaW46YWRtaW4=' -H 'Content-Type: application/json' -X POST 'http://localhost:8000/api/v1/ingest' --data-raw '[
    {
        "id": "434a5f5e-2f5f-11ed-a261-0242ac120002",
        "datetime": "24/Jun/2022:14:12:15 +0000",
        "host": "153.10.110.81",
        "soc": 1,
        "soh": 50.4,
        "temp": 1.54,
        "voltage": 1.21
    }
]'

/* Schema for test_dynamic 
1. datetime - Utf8
2. host - Utf8
3. id - Utf8
4. p_metadata - Utf8
5. p_tags - Utf8
6. p_timestamp - {"Timestamp":["Millisecond",null]}
7. soc - Int64
8. soh - Float64
9. temp - Float64
10. voltage - Float64
*/

# Invalid Request: Could not process this event due to mismatch in datatype
curl -H 'X-P-Stream: test_dynamic' -H 'Authorization: Basic YWRtaW46YWRtaW4=' -H 'Content-Type: application/json' -X POST 'http://localhost:8000/api/v1/ingest' --data-raw '[
    {
        "id": "434a5f5e-2f5f-11ed-a261-0242ac120002",
        "datetime": "24/Jun/2022:14:12:15 +0000",
        "host": "153.10.110.81",
        "soc": 1,
        "soh": 50.0,
        "temp": 1,
        "voltage": 1
    }
]'

[nitisht]

Understood @aleimu . Valid pointers! Let us discuss and get back on this please.

[de-sh]

According to the JSON spec, all numbers are the same. The problem we need to discuss is the cost of using floating point numbers as default for number type. A gist:

• UInt64 preserves exact integer values within its range, even for very large numbers.
• Float64 can represent a much wider range of numbers, but its precision is limited to about 15-17 significant decimal digits (53 bits), which can cause loss of precision for large integers.

Do we choose to default to Float64 and note this as a limitation, this since the possibility of users going the route of using a very large integer is minimal and more cautious users will anyways want to tune on costs by using static streams?