[QPROF] Feature request - Add support for central vertica

[yakirgb]

Hi Vertica team.
We using central Vertica to save dc tables from all other clusters,
All data coming to central cluster via vkconfig
It will be great if QPROF will support:

1. Add option to overwrite source schemas instead hardcoded v_internal and v_monitor .
2. Add extra filter on dc tables, for example on our central cluster, each table with extra column of _uuid for each source cluster.

Create something like:

qprof_interface = QueryProfilerInterface( 
    (63050394807998923, 1),
    key_id='yakir_1',
    target_schema='tests',
    source_schema='central_dc_collector',
    filter_on_dc_tables => "AND _uuid='some_cluster'"
    )

Thank you,
Yakir Gibraltar,
Taboola

[oualib]

can you be more precise on the options? I don't understand the use case there.

[oualib]

@yakirgb can you please send me a bigger description so I can understand what you want to achieve?

[yakirgb]

First, it's not a must, just an idea and maybe we are special on this use-case.
We saving the dc tables from all clusters into one central cluster with extra column of _uuid, we sending the data via vkconfig .
Many times we comparing performance of specific query between clusters.
It's more easy to login into one cluster and compare perfromance of some queries between clusters with one query.
So, my idea is, take all QPROF info from central cluster, just instead real dc tables schema, take the dc tables from custom schema.
And because same trx id can be on multiple clusters, add option to filter all queries by _uuid .
The benefit here it's that we saving data on central cluster for a long time and we can compare old queries while on production clusters we not saving dc table for a long time .

Thank you, Yakir Gibraltar.

[oualib]

Do you have a specific DC table in mind and a specific column so it can give me more idea.

For option one, "Add option to overwrite source schemas instead hardcoded v_internal and v_monitor" is it not already the case with the 'target_schema' and 'overwrite' parameter?

[bentzimortaboola]

@oualib since we are running multiple clusters and DC table retention is not big we are using Vertiva notify feature to send messages to kafka in parallel to write event to DC table
we have centerlize cluster with Vertica vkconfig service (consumer) that read kafka topic data and write into a REAL DC tables (so it much faster to query real tables)
As Yakir mentioned cluster identified by _uuid column, so we need a way to tell QueryProfilerInterface how to filter data