Allow return query response in columnar format

[xiangfu0]

It will be simpler for users who operate data on columnar manner without reassemble the data structure.

Could be a query option to turn it on/off.

[gortiz]

Do you mean something like returning data in arrow format? That would be awesome.

[xiangfu0]

@gortiz that would be different, I would expect that to be a different parameter like encoding

[gortiz]

I don't get it. Could you describe the task further?

Does it affect the broker and controller rest api? The GRPC direct access? The current REST format is something like:

{
  "resultTable": {
    "dataSchema": {
      "columnNames": [
        "c1",
        "c2"
      ],
      "columnDataTypes": [
        "INT",
        "STRING"
      ]
    },
    "rows": [
      [
        208,
        "a"
      ],
      [
        243,
        "b"
      ],
      [
        279,
        "c"
      ]
    ]
  }
}

Do you suggest to change it to something like:

{
  "resultTable": {
    "dataSchema": {
      "columnNames": [
        "c1",
        "c2"
      ],
      "columnDataTypes": [
        "INT",
        "STRING"
      ]
    },
    "columns": [
      [
        208,
        243,
        279
      ],
      [
        "a",
        "b",
        "c"
      ]
    ]
  },

?

That could be done with a query param, but we could also return the same data in arrow format by changing the Accept: application/json to Accept: application/vnd.apache.arrow.stream or Accept: application/vnd.apache.arrow.file

[xiangfu0]

I don't get it. Could you describe the task further?

Does it affect the broker and controller rest api? The GRPC direct access? The current REST format is something like:

{
  "resultTable": {
    "dataSchema": {
      "columnNames": [
        "c1",
        "c2"
      ],
      "columnDataTypes": [
        "INT",
        "STRING"
      ]
    },
    "rows": [
      [
        208,
        "a"
      ],
      [
        243,
        "b"
      ],
      [
        279,
        "c"
      ]
    ]
  }
}

Do you suggest to change it to something like:

{
  "resultTable": {
    "dataSchema": {
      "columnNames": [
        "c1",
        "c2"
      ],
      "columnDataTypes": [
        "INT",
        "STRING"
      ]
    },
    "columns": [
      [
        208,
        243,
        279
      ],
      [
        "a",
        "b",
        "c"
      ]
    ]
  },

?

That could be done with a query param, but we could also return the same data in arrow format by changing the Accept: application/json to Accept: application/vnd.apache.arrow.stream or Accept: application/vnd.apache.arrow.file

Exactly as you mentioned.
This is to change the response format to columnar, so java or jdbc client could use column API to iterator through data.

Arrow format is mostly for encoding support, which could be a different goal, but will definitely help for column API performance.