Skip to content
This repository has been archived by the owner on Sep 2, 2024. It is now read-only.

Asynchronously replicates Spark shuffle files in a distributed storage layer.

License

Notifications You must be signed in to change notification settings

palantir/async-shuffle-upload

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

5 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Spark Async Shuffle Upload Plugin

An implementation of SPARK-25299 that continues to write shuffle files to local disk, but asynchronously replicates shuffle files to a distributed storage layer.

More details to follow!

Gradle Tasks

./gradlew tasks - to get the list of gradle tasks

Start Developing

Run one of the following commands:

  • ./gradlew idea for IntelliJ
  • ./gradlew eclipse for Eclipse

About

Asynchronously replicates Spark shuffle files in a distributed storage layer.

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published