Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Enable features via config that are off by default in the profiler auto-tuner #577

Closed
1 task
mattahrens opened this issue Sep 21, 2023 · 7 comments · Fixed by #668
Closed
1 task
Assignees
Labels
core_tools Scope the core module (scala) feature request New feature or request

Comments

@mattahrens
Copy link
Collaborator

mattahrens commented Sep 21, 2023

Settings to consider turning on via auto-tuner recommendations based on job metrics and characteristics:

  • spark.rapids.sql.incompatibleDateFormats.enabled
@mattahrens
Copy link
Collaborator Author

@viadea: let me know if there are any other feature flag configs that are regularly tuned manually.

@mattahrens mattahrens added core_tools Scope the core module (scala) and removed ? - Needs Triage labels Sep 28, 2023
@viadea
Copy link
Collaborator

viadea commented Oct 16, 2023

Not in my memory. We do not try to ask user to enable those experimental features currently.

@parthosa
Copy link
Collaborator

parthosa commented Oct 17, 2023

This issue requires driver logs as an input to the profiler. It is tracked at #580.

@parthosa
Copy link
Collaborator

parthosa commented Nov 21, 2023

@viadea : Should these configs be recommended even if the eventlogs are not passed (but driver logs are passed)?

@parthosa
Copy link
Collaborator

parthosa commented Nov 21, 2023

  • Removing spark.rapids.sql.udfCompiler.enabled as it is no longer required.
  • Removing spark.rapids.sql.hasExtendedYearValues as it is enabled by default.

@viadea.

@viadea
Copy link
Collaborator

viadea commented Nov 21, 2023

Each of above parameters have certain limitations. I think we should suggest turn on them but also mention that customer have to refer to our doc for potential limitation

@parthosa
Copy link
Collaborator

Thanks @viadea.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core_tools Scope the core module (scala) feature request New feature or request
Projects
None yet
3 participants