[Flink] About SARG Support in Flink and BaseVector-related Issues

[ryyyyyy1]

In the planner, when passing SARG from Flink to Velox, it seems necessary to wrap the serialized string of BaseVector inside ConstantTypedExpr. This appears to require invoking the Velox4j initialization method already during the planner phase. Is this approach reasonable?

[shuai-xu]

I am not sure what do you mean, now we need not to serialize things related to native, so need not load velox4j in planner side.

[ryyyyyy1]

In planner stage, it seems we need a velox BaseVector to handler SARG in Flink, so we may need to serialize a BaseVector. or we have other better slolutions for sarg?

[shuai-xu]

Can you give an example that SARG need to pass a vector?

[ryyyyyy1]

For example, in q22, when dealing with expression lower(channel) in ('apple', 'google', 'facebook', 'baidu'). in Flink, it would be SARG['apple', 'google', 'facebook', 'baidu']. currently it will be wrapped in ConstantTypedExpr with the format of string. when passing to velox, this string format cannot be handled properly. In Velox, it seems that this kind of filter condition should be in the format of BaseVector. but i noticed that in the definition of ConstantTypedExpr, valueVector is reserved, which allow us to wrapp a serializedVector in ConstantTypedExpr, which can be handled by velox. So my idea is: when meets SARG['apple', 'google', 'facebook', 'baidu'], create a BaseVector in planner stage by calling JNI api. . serialize it and return the serializedVector string to flink and wrapped it into the ConstantTypedExpr in java. During runtime, this serializedVector string will be deserialized and handled properly.

[shuai-xu]

I haven't fully check it, but it seems ConstantTypedExpr support both variant and baseVector, and variant can tranform to Vector in velox side. Ofcourse, if really needed, we can create BaseVector in planner side.

[zhztheplayer]

Is it technically possible to initialize the Velox4J environment on client side in the Flink case? If yes I'll recommend to do this way. This is also similar to how we are doing on Gluten Spark.