Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] SQL/Task Aggregated Metrics contain duplicated executor CPU time results #1467

Open
cindyyuanjiang opened this issue Dec 17, 2024 · 1 comment
Assignees
Labels
bug Something isn't working core_tools Scope the core module (scala)

Comments

@cindyyuanjiang
Copy link
Collaborator

In Profiler tool, when aggregating SQL/Task metrics results in function aggregateSparkMetricsBySql, execCpuTime is defined and used twice in SQLTaskAggMetricsProfileResult:

We should remove this duplicated metric result.

@cindyyuanjiang cindyyuanjiang added ? - Needs Triage bug Something isn't working labels Dec 17, 2024
@amahussein amahussein added the core_tools Scope the core module (scala) label Dec 17, 2024
@amahussein
Copy link
Collaborator

Yes, I noticed that too when I was doing the optimizations in #1461
There should be only execCpuTimeSum because SQL does not have execCpuTime. It has to be aggregated on the tasks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working core_tools Scope the core module (scala)
Projects
None yet
Development

No branches or pull requests

3 participants