[FEA] Qualification tool: Add operators stats output csv file

[nartal1]

Is your feature request related to a problem? Please describe.
Currently we have rapids_4_spark_qualification_output_unsupportedOperators.csv which has details about unsupported operators and rapids_4_spark_qualification_output_execs.csv which has details about the execs. It would be good to add another csv file which has additional stats on the operators. It would help to determine the frequency of the operators in an application, task durations of the operators and so on.

Sample output format:
operator_stats.csv

AppId, SQLID, Operator_Name, Count, Total Task Exec Duration(Seconds), Impacted Stage duration, % of Stage Duration, Supported(Boolean)

where Total Task Exec Duration = sum of task exec duration which the operator is part of

Add statistics to the output file incrementally:

Tasks

Preview Give feedback

1. Qualification tool: Add output stats file for Execs(operators) #1225
   feature request user_tools +2
   
2. Add supported operators to the output file - (Stats are calculated based on unsupported, execs and stages csv file)
3. For the operators not mapped to stages, report only the totalSQLTaskduration of that SQL.

Adding stats for supported operators is a bit tricky. We have to filter out the operators if they are part of the unsupported operators. We will miss some of the operators in stats file if they are not mapped to any stages

[tgravescs]

who is the target consumer of these stats?

[nartal1]

who is the target consumer of these stats?

This is mainly for PM's and applications team. PM's can go over the statistics of what are the most frequent operators used in a job , if the unsupported operators are part of stages having larger duration and also to priortize the operators for adding support and improve the performance if they are used quite frequently.

[nartal1]

I will work on this after completing #1325. We will get more operator stats on 1325.

[nartal1]

Exec to stage-ID PR is merged #1437 and from testing we don't have much Execs that are not assigned to stages. PR to report all operators is merged too - #1444.
Closing this issue.