get_json_object(get_json_object(a, '$.b'), '$.c') => get_json_object(a, '$.b', '$.c')

印象之前是用spark rule改写成get_json_object(a, '$.b.c'), 这里为何改写成这样，这是spark function支持的形式？

改写成get_json_object(a, '$.b.c') , 有些case 兼容不了。 get_json_object(a, '$.b', '$.c') 不是spark get_json_object 兼容的形式，这里是通过ScalaUDF 的方式将get_json_object 转为 ScalaUDF，来兼容get_json_object(a, '$.b', '$.c') 这种形式。 @lgbo-ustc

什么case

1. get_json_object(get_json_object({"a":"{\"x\":5}"}', '$.a'), '$.x'), the json string has backslashes to escape quotes ;
2. get_json_object(get_json_object('{"a.b": 0}', '$.a), '$.b'), the json key contains dot character(.) and it's same as the collapsed json path;

you can also look at the comments at CollapseGetJsonObjectExpressionRule

Let's make things clear.
We cannot know the type of one field of a json. It could be a object or string. When the field's type is string, the corrent results for following cases are

So we cannot use get_json_object(s, '$.a.b') to replace get_json_object(get_json_object(s, '$.a'), '$.b'). It will cause difference with vinilla for both following choices

• keep the behavior of get_json_object(s, '$.a.b') as before, but the result is different from get_json_object(get_json_object(s, '$.a'), '$.b')
• somehow let get_json_object(s, '$.a.b') have same result as get_json_object(get_json_object(s, '$.a'), '$.b'), but it will make the result of normal call get_json_object(s, '$.a.b') different from vanilla

The optimization for nested get_json_object should be put into a seperate issue and PR, I think it will be a large change

Is the rule in #8305 disable?

rule in #8305 is disable by default as the not passed case.

I have open a seperate PR for get_json_object: #8733 @lgbo-ustc , you can have a review at this.

Hi @KevinyhZou, for get_json_object(get_json_object('{"a.b": 0}', '$.a'), '$.b'), we expect the result is still null after collapsed as get_json_object('{"a.b": 0}', '$.a.b'), right? I did a test in velox, it is null as expected (simdjson-based implementation). If CH got wrong result, I am wondering whether it is a bug in its native implementation.

Yes, it should be the problem of ch backend.