[Regression] dbt-spark-thrift docker error

[bbishopski]

Is this a regression in a recent version of dbt-spark?

• I believe this is a regression in dbt-spark functionality
• I have searched the existing issues, and I could not find an existing issue for this regression

Current Behavior

• git clone the master branch.
• docker-compose up -d
• docker container launches successfully.
• try to connect using jdbc with tool such as dbeaver or dbvisualizer or run a simple dbt test - or you can exec into the spark docker container and exec "spark-sql" and at the command prompt type "select 1;" and you will see the same error.

Notice you cannot connect and the following error is in docker logs:

ANTLR Tool version 4.7 used for code generation does not match the current runtime version 4.8ANTLR Tool version 4.7 used for code generation does not match the current runtime version 4.825/01/02 18:59:45
Exception in thread "HiveServer2-Handler-Pool: Thread-82" java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(Lorg/antlr/v4/runtime/ParserRuleContext;Lscala/Function0;)Ljava/lang/Object;
2025-01-02 11:40:04 at org.apache.spark.sql.parser.HoodieSqlCommonAstBuilder.visitSingleStatement(HoodieSqlCommonAstBuilder.scala:37)
2025-01-02 11:40:04 at org.apache.spark.sql.parser.HoodieSqlCommonAstBuilder.visitSingleStatement(HoodieSqlCommonAstBuilder.scala:31)
2025-01-02 11:40:04 at org.apache.hudi.spark.sql.parser.HoodieSqlCommonParser$SingleStatementContext.accept(HoodieSqlCommonParser.java:107)
2025-01-02 11:40:04 at org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:18)
2025-01-02 11:40:04 at org.apache.spark.sql.parser.HoodieCommonSqlParser.$anonfun$parsePlan$1(HoodieCommonSqlParser.scala:42)
2025-01-02 11:40:04 at org.apache.spark.sql.parser.HoodieCommonSqlParser.parse(HoodieCommonSqlParser.scala:84)
2025-01-02 11:40:04 at org.apache.spark.sql.parser.HoodieCommonSqlParser.parsePlan(HoodieCommonSqlParser.scala:41)
2025-01-02 11:40:04 at org.apache.spark.sql.SparkSession.$anonfun$sql$2(SparkSession.scala:620)
2025-01-02 11:40:04 at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
2025-01-02 11:40:04 at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:620)
2025-01-02 11:40:04 at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
2025-01-02 11:40:04 at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:617)
2025-01-02 11:40:04 at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:651)
2025-01-02 11:40:04 at org.apache.spark.sql.hive.thriftserver.SparkSQLSessionManager.openSession(SparkSQLSessionManager.scala:71)
2025-01-02 11:40:04 at org.apache.hive.service.cli.CLIService.openSessionWithImpersonation(CLIService.java:203)
2025-01-02 11:40:04 at org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:370)
2025-01-02 11:40:04 at org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:242)
2025-01-02 11:40:04 at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1497)
2025-01-02 11:40:04 at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1482)
2025-01-02 11:40:04 at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
2025-01-02 11:40:04 at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
2025-01-02 11:40:04 at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:52)
2025-01-02 11:40:04 at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310)
2025-01-02 11:40:04 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2025-01-02 11:40:04 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2025-01-02 11:40:04 at java.lang.Thread.run(Thread.java:750)

Expected/Previous Behavior

docker-compose up -d should launch a docker container that can be connected to!

Steps To Reproduce

• git clone the master branch.
• docker-compose up -d
• docker container launches successfully.
• try to connect using jdbc with tool such as dbeaver or dbvisualizer or run a simple dbt test - or you can exec into the spark docker container and exec "spark-sql" and at the command prompt type "select 1;" and you will see the same error.

Relevant log output

ANTLR Tool version 4.7 used for code generation does not match the current runtime version 4.8ANTLR Tool version 4.7 used for code generation does not match the current runtime version 4.825/01/02 18:59:45
Exception in thread "HiveServer2-Handler-Pool: Thread-82" java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(Lorg/antlr/v4/runtime/ParserRuleContext;Lscala/Function0;)Ljava/lang/Object;
2025-01-02 11:40:04 at org.apache.spark.sql.parser.HoodieSqlCommonAstBuilder.visitSingleStatement(HoodieSqlCommonAstBuilder.scala:37)
2025-01-02 11:40:04 at org.apache.spark.sql.parser.HoodieSqlCommonAstBuilder.visitSingleStatement(HoodieSqlCommonAstBuilder.scala:31)
2025-01-02 11:40:04 at org.apache.hudi.spark.sql.parser.HoodieSqlCommonParser$SingleStatementContext.accept(HoodieSqlCommonParser.java:107)
2025-01-02 11:40:04 at org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:18)
2025-01-02 11:40:04 at org.apache.spark.sql.parser.HoodieCommonSqlParser.$anonfun$parsePlan$1(HoodieCommonSqlParser.scala:42)
2025-01-02 11:40:04 at org.apache.spark.sql.parser.HoodieCommonSqlParser.parse(HoodieCommonSqlParser.scala:84)
2025-01-02 11:40:04 at org.apache.spark.sql.parser.HoodieCommonSqlParser.parsePlan(HoodieCommonSqlParser.scala:41)
2025-01-02 11:40:04 at org.apache.spark.sql.SparkSession.$anonfun$sql$2(SparkSession.scala:620)
2025-01-02 11:40:04 at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
2025-01-02 11:40:04 at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:620)
2025-01-02 11:40:04 at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
2025-01-02 11:40:04 at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:617)
2025-01-02 11:40:04 at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:651)
2025-01-02 11:40:04 at org.apache.spark.sql.hive.thriftserver.SparkSQLSessionManager.openSession(SparkSQLSessionManager.scala:71)
2025-01-02 11:40:04 at org.apache.hive.service.cli.CLIService.openSessionWithImpersonation(CLIService.java:203)
2025-01-02 11:40:04 at org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:370)
2025-01-02 11:40:04 at org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:242)
2025-01-02 11:40:04 at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1497)
2025-01-02 11:40:04 at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1482)
2025-01-02 11:40:04 at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
2025-01-02 11:40:04 at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
2025-01-02 11:40:04 at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:52)
2025-01-02 11:40:04 at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310)
2025-01-02 11:40:04 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2025-01-02 11:40:04 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2025-01-02 11:40:04 at java.lang.Thread.run(Thread.java:750)

### Environment

```markdown
- OS: mac os
- Python:
- dbt-core (working version): unrelated
- dbt-spark (working version): unknown, but I assume this was working at some point
- dbt-core (regression version): unrelated
- dbt-spark (regression version): master branch docker compose file

Additional Context

Looking at spark 3.3.2 docker file, it looks like java 11 is required?

https://github.com/apache/spark-docker/blob/master/3.3.2/scala2.12-java11-python3-ubuntu/Dockerfile

However, when I replaced dbt-spark spark.Dockerfile with 11, I still got the same error.

To fix this I had to completely replace the spark.Dockerfile with the spark docker file (above) contents, and also add the postgres jar as in original spark.Dockerfile.

After using the "modified" https://github.com/apache/spark-docker/blob/master/3.3.2/scala2.12-java11-python3-ubuntu/Dockerfile, it worked as expected. Here's my modified docker file (I added the postgres and also set PATH)

Licensed to the Apache Software Foundation (ASF) under one or more

contributor license agreements. See the NOTICE file distributed with

this work for additional information regarding copyright ownership.

The ASF licenses this file to You under the Apache License, Version 2.0

(the "License"); you may not use this file except in compliance with

the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software

distributed under the License is distributed on an "AS IS" BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

See the License for the specific language governing permissions and

limitations under the License.

FROM eclipse-temurin:11-jre-focal

ARG spark_uid=185

RUN groupadd --system --gid=${spark_uid} spark &&
useradd --system --uid=${spark_uid} --gid=spark spark

RUN set -ex &&
apt-get update &&
apt-get install -y wget netcat-openbsd procps libpostgresql-jdbc-java &&
ln -s /lib /lib64 &&
apt install -y gnupg2 wget bash tini libc6 libpam-modules krb5-user libnss3 procps net-tools gosu &&
apt install -y python3 python3-pip &&
mkdir -p /opt/spark &&
mkdir /opt/spark/python &&
mkdir -p /opt/spark/examples &&
mkdir -p /opt/spark/work-dir &&
touch /opt/spark/RELEASE &&
chown -R spark:spark /opt/spark &&
rm /bin/sh &&
ln -sv /bin/bash /bin/sh &&
echo "auth required pam_wheel.so use_uid" >> /etc/pam.d/su &&
chgrp root /etc/passwd && chmod ug+rw /etc/passwd &&
rm -rf /var/cache/apt/* &&
rm -rf /var/lib/apt/lists/*

Install Apache Spark

https://downloads.apache.org/spark/KEYS

ENV SPARK_TGZ_URL=https://archive.apache.org/dist/spark/spark-3.3.2/spark-3.3.2-bin-hadoop3.tgz
SPARK_TGZ_ASC_URL=https://archive.apache.org/dist/spark/spark-3.3.2/spark-3.3.2-bin-hadoop3.tgz.asc
GPG_KEY=C56349D886F2B01F8CAE794C653C2301FEA493EE

RUN set -ex;
export SPARK_TMP="$(mktemp -d)";
cd $SPARK_TMP;
wget -nv -O spark.tgz "$SPARK_TGZ_URL";
wget -nv -O spark.tgz.asc "$SPARK_TGZ_ASC_URL";
export GNUPGHOME="$(mktemp -d)";
gpg --keyserver hkps://keys.openpgp.org --recv-key "$GPG_KEY" ||
gpg --keyserver hkps://keyserver.ubuntu.com --recv-keys "$GPG_KEY";
gpg --batch --verify spark.tgz.asc spark.tgz;
gpgconf --kill all;
rm -rf "$GNUPGHOME" spark.tgz.asc;

tar -xf spark.tgz --strip-components=1;
chown -R spark:spark .;
mv jars /opt/spark/;
mv bin /opt/spark/;
mv sbin /opt/spark/;
mv kubernetes/dockerfiles/spark/decom.sh /opt/;
mv examples /opt/spark/;
mv kubernetes/tests /opt/spark/;
mv data /opt/spark/;
mv python/pyspark /opt/spark/python/pyspark/;
mv python/lib /opt/spark/python/lib/;
cd ..;
rm -rf "$SPARK_TMP";
ln -s /usr/share/java/postgresql-jdbc4.jar /opt/spark/jars/postgresql-jdbc4.jar;

COPY entrypoint.sh /opt/

ENV SPARK_HOME /opt/spark
ENV PATH="$PATH:/opt/spark/bin"

WORKDIR /opt/spark/work-dir
RUN chmod g+w /opt/spark/work-dir
RUN chmod a+x /opt/decom.sh
RUN chmod a+x /opt/entrypoint.sh

ENTRYPOINT [ "/opt/entrypoint.sh" ]