Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CORE] SparkPlan without logicalLink does not show correct fallback reason on UI #8596

Open
zml1206 opened this issue Jan 23, 2025 · 2 comments
Labels
bug Something isn't working triage

Comments

@zml1206
Copy link
Contributor

zml1206 commented Jan 23, 2025

Bug description

Currently, set the fallback reason to logical plan

private def printFallbackReason(plan: SparkPlan): Unit = {
val validationLogLevel = glutenConf.validationLogLevel
plan.foreachUp {
case _: GlutenPlan => // ignore
case p: SparkPlan if FallbackTags.nonEmpty(p) =>
val tag = FallbackTags.get(p)
logFallbackReason(validationLogLevel, p.nodeName, tag.reason())
// With in next round stage in AQE, the physical plan would be a new instance that
// can not preserve the tag, so we need to set the fallback reason to logical plan.
// Then we can be aware of the fallback reason for the whole plan.
// If a logical plan mapping to several physical plan, we add all reason into
// that logical plan to make sure we do not lose any fallback reason.
p.logicalLink.foreach(logicalPlan => FallbackTags.add(logicalPlan, tag))
case _ =>
}
}
.
There is a problem with this, SparkPlan without logicalLink does not show correct fallback reason on UI.
Some additional nodes may be generated in gluten's rules, such as project generated by PushDownInputFileExpression.

Spark version

None

Spark configurations

No response

System information

No response

Relevant logs

@zml1206 zml1206 added bug Something isn't working triage labels Jan 23, 2025
@zml1206
Copy link
Contributor Author

zml1206 commented Jan 23, 2025

@ulysses-you AQE will also trigger the gluten rule again and retag for new physical plan. Is it necessary to set the fallback reason to logical plan, if so, can you explain the reason in detail, thank you?

@ulysses-you
Copy link
Contributor

cc @wForget I remember you asked similar quetion before. do you have any update ?

@FelixYBW FelixYBW changed the title SparkPlan without logicalLink does not show correct fallback reason on UI [VL] SparkPlan without logicalLink does not show correct fallback reason on UI Jan 23, 2025
@zml1206 zml1206 changed the title [VL] SparkPlan without logicalLink does not show correct fallback reason on UI [CORE] SparkPlan without logicalLink does not show correct fallback reason on UI Jan 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working triage
Projects
None yet
Development

No branches or pull requests

2 participants