Skip to content

Commit bd37b10

Browse files
GauthamBanasandradongjoon-hyun
authored andcommitted
[SPARK-41377][BUILD] Fix spark-version-info.properties not found on Windows
### What changes were proposed in this pull request? This PR enhances the Maven build configuration to automatically detect and switch between using Powershell for Windows and Bash for non-Windows OS to generate `spark-version-info.properties` file. ### Why are the changes needed? While building Spark, the `spark-version-info.properties` file [is generated using bash](https://github.com/apache/spark/blob/d62c18b7497997188ec587e1eb62e75c979c1c93/core/pom.xml#L560-L564). In Windows environment, if Windows Subsystem for Linux (WSL) is installed, it somehow overrides the other bash executables in the PATH, as noted in SPARK-40739. The bash in WSL has a different mounting configuration and thus, [the target location specified for spark-version-info.properties](https://github.com/apache/spark/blob/d62c18b7497997188ec587e1eb62e75c979c1c93/core/pom.xml#L561-L562) won't be the expected location. Ultimately, this leads to `spark-version-info.properties` to get excluded from the spark-core jar, thus causing the SparkContext initialization to fail with the above depicted error message. This PR fixes the issue by directing the build system to use the right shell according to the platform. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? I tested this by building on a Windows 10 PC. ```psh mvn -Pyarn '-Dhadoop.version=3.3.0' -DskipTests clean package ``` Once the build finished, I verified that `spark-version-info.properties` file was included in the spark-core jar. ![image](https://user-images.githubusercontent.com/10280768/205497898-80e53617-c991-460e-b04a-a3bdd4f298ae.png) I also ran the SparkPi application and verified that it ran successfully without any errors. ![image](https://user-images.githubusercontent.com/10280768/205499567-f6e8e10a-dcbb-45fb-b282-fc29ba58adee.png) Closes apache#38903 from GauthamBanasandra/spark-version-info-ps. Authored-by: Gautham Banasandra <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 3212fa9 commit bd37b10

File tree

3 files changed

+79
-2
lines changed

3 files changed

+79
-2
lines changed

appveyor.yml

+1
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ only_commits:
2828
files:
2929
- appveyor.yml
3030
- dev/appveyor-install-dependencies.ps1
31+
- build/spark-build-info.ps1
3132
- R/
3233
- sql/core/src/main/scala/org/apache/spark/sql/api/r/
3334
- core/src/main/scala/org/apache/spark/api/r/

build/spark-build-info.ps1

+46
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
#
2+
# Licensed to the Apache Software Foundation (ASF) under one or more
3+
# contributor license agreements. See the NOTICE file distributed with
4+
# this work for additional information regarding copyright ownership.
5+
# The ASF licenses this file to You under the Apache License, Version 2.0
6+
# (the "License"); you may not use this file except in compliance with
7+
# the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
18+
# This script generates the build info for spark and places it into the spark-version-info.properties file.
19+
# Arguments:
20+
# ResourceDir - The target directory where properties file would be created. [./core/target/extra-resources]
21+
# SparkVersion - The current version of spark
22+
23+
param(
24+
# The resource directory.
25+
[Parameter(Position = 0)]
26+
[String]
27+
$ResourceDir,
28+
29+
# The Spark version.
30+
[Parameter(Position = 1)]
31+
[String]
32+
$SparkVersion
33+
)
34+
35+
$null = New-Item -Type Directory -Force $ResourceDir
36+
$SparkBuildInfoPath = $ResourceDir.TrimEnd('\').TrimEnd('/') + '\spark-version-info.properties'
37+
38+
$SparkBuildInfoContent =
39+
"version=$SparkVersion
40+
user=$($Env:USERNAME)
41+
revision=$(git rev-parse HEAD)
42+
branch=$(git rev-parse --abbrev-ref HEAD)
43+
date=$([DateTime]::UtcNow | Get-Date -UFormat +%Y-%m-%dT%H:%M:%SZ)
44+
url=$(git config --get remote.origin.url)"
45+
46+
Set-Content -Path $SparkBuildInfoPath -Value $SparkBuildInfoContent

core/pom.xml

+32-2
Original file line numberDiff line numberDiff line change
@@ -558,12 +558,42 @@
558558
<artifactId>maven-antrun-plugin</artifactId>
559559
<executions>
560560
<execution>
561+
<id>choose-shell-and-script</id>
562+
<phase>validate</phase>
563+
<goals>
564+
<goal>run</goal>
565+
</goals>
566+
<configuration>
567+
<exportAntProperties>true</exportAntProperties>
568+
<target>
569+
<condition property="shell" value="powershell.exe" else="bash">
570+
<and>
571+
<os family="windows"/>
572+
</and>
573+
</condition>
574+
<condition property="spark-build-info-script" value="spark-build-info.ps1"
575+
else="spark-build-info">
576+
<and>
577+
<os family="windows"/>
578+
</and>
579+
</condition>
580+
<echo>Shell to use for generating spark-version-info.properties file =
581+
${shell}
582+
</echo>
583+
<echo>Script to use for generating spark-version-info.properties file =
584+
${spark-build-info-script}
585+
</echo>
586+
</target>
587+
</configuration>
588+
</execution>
589+
<execution>
590+
<id>generate-spark-build-info</id>
561591
<phase>generate-resources</phase>
562592
<configuration>
563593
<!-- Execute the shell script to generate the spark build information. -->
564594
<target>
565-
<exec executable="bash">
566-
<arg value="${project.basedir}/../build/spark-build-info"/>
595+
<exec executable="${shell}">
596+
<arg value="${project.basedir}/../build/${spark-build-info-script}"/>
567597
<arg value="${project.build.directory}/extra-resources"/>
568598
<arg value="${project.version}"/>
569599
</exec>

0 commit comments

Comments
 (0)