Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support parsing of inprogress eventlogs #686

Merged
merged 4 commits into from
Dec 13, 2023
Merged

Support parsing of inprogress eventlogs #686

merged 4 commits into from
Dec 13, 2023

Conversation

amahussein
Copy link
Collaborator

@amahussein amahussein commented Dec 12, 2023 •
edited
Loading

Signed-off-by: Ahmed Hussein (amahussein) [email protected]

Fixes #685

  • Added feature: support parsing of "inprogress" eventlogs
  • Catches exception thrown by Json Parser when an unexpected EOF occurs
  • Added a warning message when the eventlog has unexpected EOF (incomplete events)
  • Added UT to verify that an incomplete eventlog can be parsed successfully

Before Changes:

For an incomplete eventlog (i.e., eventlog.inprogress), the "app" resulting from processing the eventlog is skipped. It is considered as "N/A".

After Changes:

The "app" is considered successful.

@amahussein
Support parsing of inprogress eventlogs
5c4b67f 
Signed-off-by: Ahmed Hussein (amahussein) <[email protected]>

Fixes NVIDIA#685

* Added feature: support parsing of "inprogress" eventlogs
* Catches exception thrown by Json Parser when an unexpected EOF occurs
@amahussein amahussein added bug Something isn't working core_tools Scope the core module (scala) labels Dec 12, 2023
@amahussein amahussein self-assigned this Dec 12, 2023
@amahussein
Add fix for Spark-3.4.1+
7674680 
Signed-off-by: Ahmed Hussein (amahussein) <[email protected]>
@amahussein
Remove unused imports from QualificationSuite
98c825d 
Signed-off-by: Ahmed Hussein (amahussein) <[email protected]>
@amahussein
address code review
6952d6e 
Signed-off-by: Ahmed Hussein (amahussein) <[email protected]>
@amahussein amahussein requested a review from parthosa December 13, 2023 15:45
parthosa
parthosa approved these changes Dec 13, 2023
View reviewed changes
Copy link
Collaborator

@parthosa parthosa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @amahussein

nartal1
nartal1 approved these changes Dec 13, 2023
View reviewed changes
Copy link
Collaborator

@nartal1 nartal1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @amahussein !

@amahussein amahussein merged commit e298c53 into NVIDIA:dev Dec 13, 2023
9 checks passed
@amahussein amahussein deleted the spark-rapids-tools-685 branch December 13, 2023 20:52
@amahussein amahussein mentioned this pull request Jun 14, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nartal1 nartal1 nartal1 approved these changes

@parthosa parthosa parthosa approved these changes

@tgravescs tgravescs Awaiting requested review from tgravescs

@mattahrens mattahrens Awaiting requested review from mattahrens

Assignees

@amahussein amahussein

Labels
bug Something isn't working core_tools Scope the core module (scala)
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[BUG] Profiler ignores applications in incomplete eventlogs
3 participants
@amahussein @parthosa @nartal1