PPL: Add `json` function and `cast(x as json)` function #3243

acarbonetto · 2025-01-13T18:03:45Z

Description

For OpenSearch-PPL, adds the json() and cast(expression as json) functions. Both of these functions can be used to convert a json-encoded string into an expression object. These functions return any expression type that is valid json (object, array, or scalar). Unfortunately, because the return type is unknown, the expression's return type is UNDEFINED.

The user can use cast to convert the types where necessary for scalar types. For example:

// converts json-encoded integer to an int expression
cast(json(json_string) as int)

The following casts are available to convert json-encoded scalars: int, long, double, float, boolean.

JSON
----------

Description
>>>>>>>>>>>

Usage: `json(value)` Evaluates whether a string can be parsed as a json-encoded string and casted as an expression. Returns the JSON value if valid, null otherwise.

Argument type: STRING

Return type: BOOLEAN/DOUBLE/INTEGER/NULL/STRUCT/ARRAY

Example::

    > source=json_test | where json_valid(json_string) | eval json=json(json_string) | fields test_name, json_string, json
    fetched rows / total rows = 4/4
    +---------------------+------------------------------+---------------+
    | test_name           | json_string                  | json          |
    |---------------------|------------------------------|---------------|
    | json object         | {"a":"1","b":"2"}            | {a:"1",b:"2"} |
    | json array          | [1, 2, 3, 4]                 | [1,2,3,4]     |
    | json scalar string  | "abc"                        | "abc"         |
    | json empty string   |                              | null          |
    +---------------------+------------------------------+---------------+

Related Issues

Resolves #3209

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

core/src/main/java/org/opensearch/sql/expression/json/JsonFunctions.java

core/src/main/java/org/opensearch/sql/expression/operator/convert/TypeCastOperators.java

core/src/main/java/org/opensearch/sql/utils/JsonUtils.java

core/src/test/java/org/opensearch/sql/expression/json/JsonFunctionsTest.java

Signed-off-by: Kenrick Yap <[email protected]>

Signed-off-by: Andrew Carbonetto <[email protected]>

docs/user/ppl/functions/json.rst

core/src/test/java/org/opensearch/sql/expression/datetime/DateTimeFunctionTest.java

core/src/test/java/org/opensearch/sql/expression/datetime/YearweekTest.java

kenrickyap

LGTM minor nits

Signed-off-by: Andrew Carbonetto <[email protected]>

docs/user/ppl/functions/json.rst

Signed-off-by: Andrew Carbonetto <[email protected]>

YANG-DB · 2025-01-28T19:21:17Z

core/src/main/java/org/opensearch/sql/utils/JsonUtils.java

I think a license header is missing...

Will add.
I'm surprised that it isn't added automagically by spotless.

YANG-DB · 2025-01-28T19:24:31Z

core/src/test/java/org/opensearch/sql/expression/json/JsonFunctionsTest.java

-    FunctionExpression exp = DSL.jsonValid(DSL.literal(jsonString));
-    return exp.valueOf();
+  @Test
+  void json_returnsSemanticCheckException() {


can you add a few more failed tests use cases here ?

YANG-DB · 2025-01-28T19:29:40Z

integ-test/src/test/resources/json_test.json

@@ -1,14 +1,24 @@
 {"index":{"_id":"0"}}
-{"test_name":"json nested object", "json_string":"{\"a\":\"1\",\"b\":{\"c\":\"2\",\"d\":\"3\"}}"}
+{"test_name":"json nested object", "json_string":"{\"a\":\"1\", \"b\": {\"c\": \"3\"}, \"d\": [1, 2, 3]}"}


can u also add a use case for inner json obj that has an array ?
{"a":{"b":{"c":[....]}}}

"d" is mapped to an array. Do you want an object within in an array?

YANG-DB

IMO looking good - added some minor comments...

Signed-off-by: Andrew Carbonetto <[email protected]>

acarbonetto · 2025-01-29T18:22:34Z

IMO looking good - added some minor comments...

Tests added. Thank you for looking.

YANG-DB · 2025-01-30T23:30:11Z

@acarbonetto can u plz resolve the conflicts ?
thanks

acarbonetto requested review from ps48, kavithacm, derek-ho, joshuali925, dai-chen, YANG-DB, mengweieric, Swiddis, penghuo, seankao-az, MaxKsyunz, Yury-Fridlyand, anirudha, forestmvey, GumpacG, ykmr1224, LantaoJin and noCharger as code owners January 13, 2025 18:03

acarbonetto force-pushed the feature/acarbo_json_cast_ppl branch 2 times, most recently from 2ccf4ab to 3ec16e0 Compare January 15, 2025 18:33