[GLUTEN-6893][VL] Change to using native libs generated by vcpkg buil…

…d in Gluten scala tests (#6894)
apache · Aug 21, 2024 · aab550f · aab550f
1 parent 11a236d
commit aab550f
Show file tree

Hide file tree

Showing 8 changed files with 129 additions and 117 deletions.
diff --git a/.github/workflows/velox_docker.yml b/.github/workflows/velox_docker.yml
diff --git a/dev/ci-velox-buildshared-centos-8.sh b/dev/ci-velox-buildshared-centos-8.sh
@@ -10,9 +10,8 @@ yum install sudo patch java-1.8.0-openjdk-devel wget -y
 wget https://downloads.apache.org/maven/maven-3/3.8.8/binaries/apache-maven-3.8.8-bin.tar.gz
 tar -xvf apache-maven-3.8.8-bin.tar.gz && mv apache-maven-3.8.8 /usr/lib/maven
 export PATH="${PATH}:/usr/lib/maven/bin"
+echo "PATH=${PATH}:/usr/lib/maven/bin" >> $GITHUB_ENV
 
 source /opt/rh/gcc-toolset-9/enable
 ./dev/builddeps-veloxbe.sh --run_setup_script=OFF --enable_ep_cache=OFF --build_tests=ON \
     --build_examples=ON --build_benchmarks=ON --build_protobuf=ON
-
-cd ./cpp/build && ctest -V
diff --git a/dev/ci-velox-buildstatic-centos-7.sh b/dev/ci-velox-buildstatic-centos-7.sh
@@ -3,8 +3,6 @@
 set -e
 
 source /opt/rh/devtoolset-9/enable
-cd $GITHUB_WORKSPACE/
-source ./dev/vcpkg/env.sh
 export NUM_THREADS=4
-./dev/builddeps-veloxbe.sh --build_tests=OFF --build_benchmarks=OFF --build_arrow=OFF --enable_s3=ON \
-                           --enable_gcs=ON --enable_hdfs=ON --enable_abfs=ON
+./dev/builddeps-veloxbe.sh --enable_vcpkg=ON --build_arrow=OFF --build_tests=OFF --build_benchmarks=OFF \
+                           --build_examples=OFF --enable_s3=ON --enable_gcs=ON --enable_hdfs=ON --enable_abfs=ON
diff --git a/dev/vcpkg/ports/simdjson/vcpkg.json b/dev/vcpkg/ports/simdjson/vcpkg.json
@@ -17,8 +17,7 @@
   "default-features": [
     "deprecated",
     "exceptions",
-    "threads",
-    "utf8-validation"
+    "threads"
   ],
   "features": {
     "deprecated": {

diff --git a/docs/developers/UsingGperftoolsInCH.md b/docs/developers/UsingGperftoolsInCH.md
@@ -1,3 +1,10 @@
+---
+layout: page
+title: Debug for CH Backend with gpertools Tool
+nav_order: 11
+has_children: true
+parent: /developer-overview/
+---
 We need using gpertools to find the memory or CPU issue. That's what this document is about.
 
 ## Install gperftools

diff --git a/docs/developers/UsingJemallocWithCH.md b/docs/developers/UsingJemallocWithCH.md
@@ -1,3 +1,10 @@
+---
+layout: page
+title: Use Jemalloc for CH Backend
+nav_order: 12
+has_children: true
+parent: /developer-overview/
+---
 We need using jemalloc to find the memory issue. That's what this document is about.
 
 ## Change code of jemalloc

diff --git a/docs/developers/VeloxUDF.md b/docs/developers/VeloxUDF.md
@@ -1,3 +1,10 @@
+---
+layout: page
+title: Velox UDF and UDAF
+nav_order: 13
+has_children: true
+parent: /developer-overview/
+---
 # Velox User-Defined Functions (UDF) and User-Defined Aggregate Functions (UDAF)
 
 ## Introduction

diff --git a/docs/developers/velox-backend-CI.md b/docs/developers/velox-backend-CI.md
@@ -0,0 +1,24 @@
+---
+layout: page
+title: Velox Backend CI
+nav_order: 14
+parent: Developer Overview
+---
+# Velox Backend CI
+
+GHA workflows are defined under `.github/workflows/`.
+
+## Docker Build
+We have a weekly job to build a docker based on `Dockerfile.gha` for CI verification, defined in `docker_image.yml`.
+
+## Vcpkg Caching
+Gluten main branch is pulled down during docker build. And vcpkg will cache binary data of all dependencies defined under dev/vcpkg.
+These binary data is cached into `/var/cache/vcpkg` and CI job can re-use them in new build. By setting `VCPKG_BINARY_SOURCES=clear` in env.,
+reusing vcpkg cache can be disabled.
+
+## Arrow Libs Pre-installation
+Arrow libs are pre-installed in docker, assuming they are not actively changed, then not necessarily to be re-built every time.
+
+## Updating Docker Image
+Two GitHub secrets `DOCKERHUB_USER` & `DOCKERHUB_TOKEN` can be used to push docker image to docker hub: https://hub.docker.com/r/apache/gluten/tags.
+Note GitHub secrets are not retrievable in PR from forked repo.