Skip to content

Commit

Permalink
[GLUTEN-6876][VL] update spark 3.5.2 in doc (#8543)
Browse files Browse the repository at this point in the history
* Update Velox.md

* Update build-guide.md

* Update Velox.md
  • Loading branch information
FelixYBW authored Jan 16, 2025
1 parent d286e12 commit ddf7e32
Show file tree
Hide file tree
Showing 2 changed files with 5 additions and 5 deletions.
8 changes: 4 additions & 4 deletions docs/get-started/Velox.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ parent: Getting-Started

| Type | Version |
|-------|------------------------------|
| Spark | 3.2.2, 3.3.1, 3.4.3, 3.5.1 |
| Spark | 3.2.2, 3.3.1, 3.4.3, 3.5.2 |
| OS | Ubuntu20.04/22.04, Centos7/8 |
| jdk | openjdk8/jdk17 |
| scala | 2.12 |
Expand All @@ -18,7 +18,7 @@ parent: Getting-Started

Currently, with static build Gluten+Velox backend supports all the Linux OSes, but is only tested on **Ubuntu20.04/Ubuntu22.04/Centos7/Centos8**. With dynamic build, Gluten+Velox backend support **Ubuntu20.04/Ubuntu22.04/Centos7/Centos8** and their variants.

Currently, the officially supported Spark versions are 3.2.2, 3.3.1, 3.4.3 and 3.5.1.
Currently, the officially supported Spark versions are 3.2.2, 3.3.1, 3.4.3 and 3.5.2.

We need to set up the `JAVA_HOME` env. Currently, Gluten supports **java 8** and **java 17**.

Expand Down Expand Up @@ -113,11 +113,11 @@ Alternatively you may refer to [build in docker](docs/developers/velox-backend-b
With build option `enable_vcpkg=ON`, all dependency libraries will be statically linked to `libvelox.so` and `libgluten.so` which are packed into the gluten-jar.
In this way, only the gluten-jar is needed to add to `spark.<driver|executor>.extraClassPath` and spark will deploy the jar to each worker node. It's better to build
the static version using a clean docker image without any extra libraries installed ( [build in docker](docs/developers/velox-backend-build-in-docker.md) ). On host with
some libraries like jemalloc installed, the script may crash with odd message. You may need to uninstall those libraries to get a clean host. We ** strongly recommend ** user to build Gluten in this way to avoid dependency lacking issue.
some libraries like jemalloc installed, the script may crash with odd message. You may need to uninstall those libraries to get a clean host. We **strongly recommend** user to build Gluten in this way to avoid dependency lacking issue.

With build option `enable_vcpkg=OFF`, not all dependency libraries will be dynamically linked. After building, you need to separately execute `./dev/build-thirdparty.sh` to
pack required shared libraries into another jar named `gluten-thirdparty-lib-$LINUX_OS-$VERSION-$ARCH.jar`. Then you need to add the jar to Spark config `extraClassPath` and
set `spark.gluten.loadLibFromJar=true`. Otherwise, you need to install required shared libraries with ** exactly the same versions ** on each worker node . You may find the
set `spark.gluten.loadLibFromJar=true`. Otherwise, you need to install required shared libraries with **exactly the same versions** on each worker node . You may find the
libraries list from the third-party jar.

# Remote storage support
Expand Down
2 changes: 1 addition & 1 deletion docs/get-started/build-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,4 +74,4 @@ It's name pattern is `gluten-<backend_type>-bundle-spark<spark.bundle.version>_<
| 3.2.2 | 3.2 | 2.12 |
| 3.3.1 | 3.3 | 2.12 |
| 3.4.3 | 3.4 | 2.12 |
| 3.5.1 | 3.5 | 2.12 |
| 3.5.2 | 3.5 | 2.12 |

0 comments on commit ddf7e32

Please sign in to comment.