Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Release builds for newer versions of Scala and Spark #84

Open
colindean opened this issue Jun 2, 2022 · 3 comments
Open

Release builds for newer versions of Scala and Spark #84

colindean opened this issue Jun 2, 2022 · 3 comments
Labels
enhancement New feature or request hacktoberfest Good issues for Hacktoberfest

Comments

@colindean
Copy link
Collaborator

Is your feature request related to a problem? Please describe.

We currently only ship for Scala 2.11 and Spark 2.3.x.

Describe the solution you'd like

We should ship for newer versions in whatever pairs are appropriate.

Additional context

https://github.com/sbt/sbt-projectmatrix is probably the right tool for the job.
@colindean colindean added the enhancement New feature or request label Jun 2, 2022
@colindean colindean mentioned this issue Jun 7, 2022
Merged
@colindean
Copy link
Collaborator Author

Minimal:

groups:
  - spark:
    - 2.3.4
    scala:
    - 2.11.12

Max:

groups:
  - spark:
    - 2.3.4
    scala:
    - 2.11.12
  - spark:
    - 2.4.8
    - 3.0.3
    - 3.1.3
    scala:
    - 2.12.15
  - spark:
    - 3.2.1
    scala:
    - 2.13.8

Spark versions: https://spark.apache.org/downloads.html

@colindean
Copy link
Collaborator Author

We should continue to build on JDK 1.8. Spark still supports that. However, Spark 3.0 allows JDK 1.11, so that could be another variable, but it'd have to be at the GHA level instead of sbt-projectmatrix.

colindean added a commit that referenced this issue Jun 9, 2022
@colindean
Implements basic crossbuild for 2.{11,12,13}
75375e3 
Done:

* Compile and tests for 2.11
* Assembly for 2.11

TODO:

* Compile and tests for 2.12 and 2.13
* Assembly for 2.12 and 2.13

Work toward #84
@colindean
Copy link
Collaborator Author

I've got a start on this in branch gh-84-crossbuild. I've got builds for [Scala 2.11 + Spark 2.3.4] and [Scala 2.12 + Spark 2.4.8] working pretty easily but not Scala 2.13 yet. It'll need some refactoring and I've yet to address a spurious extra build, too.

@colindean colindean mentioned this issue Jun 14, 2022
Open
@colindean colindean mentioned this issue Aug 16, 2022
Closed
colindean added a commit that referenced this issue Sep 16, 2022
@colindean
Merge remote-tracking branch 'origin/master' into gh-84-crossbuild
06e54f5
@colindean colindean added the hacktoberfest Good issues for Hacktoberfest label Oct 3, 2022
colindean added a commit that referenced this issue Apr 3, 2023
@colindean
Enables lockfile workflow with sbt-dependency-lock
aa1f1f7 
Caveats of this:

1. [Crossbuilds may not be supported][crossbuilds], impacting #84.
2. The hash used is SHA1, which is deprecated including by [NIST][nist].
   Some work is in draft PRs for introducing a new version of the
   lockfile that may enable SHA256 or other hash algorithms.

Fixes #129

[crossbuilds]: stringbean/sbt-dependency-lock#13
[nist]: https://www.nist.gov/news-events/news/2022/12/nist-retires-sha-1-cryptographic-algorithm
@colindean colindean mentioned this issue Apr 3, 2023
Draft
@colindean colindean mentioned this issue Jun 15, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request hacktoberfest Good issues for Hacktoberfest
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@colindean