Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Unit-test csv for unsupported operators fails on spark340 #498

Closed
amahussein opened this issue Aug 18, 2023 · 0 comments · Fixed by #503
Closed

[BUG] Unit-test csv for unsupported operators fails on spark340 #498

amahussein opened this issue Aug 18, 2023 · 0 comments · Fixed by #503
Assignees
@amahussein
Labels
bug Something isn't working core_tools Scope the core module (scala)

Comments

@amahussein
Copy link
Collaborator

Describe the bug

While testing #471 , I found that the build fails for spark-340

Basically we need to update delta-lake dependency from 2.4.0rc1 to 2.4.0

then there is a failed unit test

- test csv output for unsupported operators *** FAILED ***
  Stream("App ID,Unsupported Type,Details,Notes", ""local-1692378607246","Read","JSON","Types not supported - bigint:int"", ""local-1692378607246","Exec","AdaptiveSparkPlan",""", ""local-1692378607246","Exec","WriteFiles",""", ""local-1692378607246","Exec","Execute InsertIntoHadoopFsRelationCommand json",""", ""local-1692378607246","Exec","LocalTableScan",""", ""local-1692378607246","Exec","Scan json",""", ""local-1692378607246","Write","json",""") had size 8 instead of expected size 6 (QualificationSuite.scala:1157)

Steps/Code to reproduce bug

mvn -Dbuildver=340 package
@amahussein amahussein added bug Something isn't working ? - Needs Triage labels Aug 18, 2023
amahussein added a commit to amahussein/spark-rapids-tools that referenced this issue Aug 22, 2023
@amahussein
Fix unit-tests for Spark-340
56db281 
Signed-off-by: Ahmed Hussein (amahussein) <[email protected]>

Fixes NVIDIA#498, Fixes NVIDIA#463

- Update the gh-workflow to run mvn for list of spark-versions
- Fixes the delta-lake dependency in the pom.xml file
- Fix the broken unit test for spark-340
- Update the spark-341 to point to release instead of snapshot
- Fix the Autotuner warning
@amahussein amahussein mentioned this issue Aug 22, 2023
Merged
@amahussein amahussein self-assigned this Aug 22, 2023
@amahussein amahussein added the core_tools Scope the core module (scala) label Aug 22, 2023
amahussein added a commit that referenced this issue Aug 22, 2023
@amahussein
Fix unit-tests for Spark-340 and Add spark-versions to gh-workflow (#503
49d1470 
)

* Fix unit-tests for Spark-340

Fixes #498, Fixes #463

- Update the gh-workflow to run mvn for list of spark-versions
- Fixes the delta-lake dependency in the pom.xml file
- Fix the broken unit test for spark-340
- Update the spark-341 to point to release instead of snapshot
- Fix the Autotuner warning

* Fix the unit-test for spark-320-330

---------

Signed-off-by: Ahmed Hussein (amahussein) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@amahussein amahussein

Labels
bug Something isn't working core_tools Scope the core module (scala)
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix unit-tests for Spark-340 and Add spark-versions to gh-workflow amahussein/spark-rapids-tools
1 participant
@amahussein