Running pytest with local spark session

[rotemb-cye]

Hey,

I am trying to run pytest on my local PC, when databricks extension is installed.
I am trying to create local spark session:

def get_spark_session():
    spark = (
        SparkSession.builder.master("local[*]")
        .appName("local-tests")
        .config("spark.driver.bindAddress", "127.0.0.1")
        .getOrCreate()
    )
    return spark


@pytest.mark.etl
@pytest.fixture(scope="session")
def spark_session():
    spark = get_spark_session()
    yield spark
    spark.stop()

and I get the following error:
RuntimeError: Only remote Spark sessions using Databricks Connect are supported. Could not find connection parameters to start a Spark remote session.

How to solve it? I want to be able to run my pytest when being offline

TNXXX

[htuomola]

Hi, this is the same exact issue that we have been struggling with. It seems that installing databricks-connect modifies installed pyspark package and adds throwing this error to the code. I'm also interested in finding a workaround for this because in the current state it basically blocks using Databricks Connect.

[benoitLebreton-perso]

Hello I managed to get my local spark session working by the following VSCode command palette

In fact, even uninstalling the extension was not working.

[odimko]

thanks for your solution @benoitLebreton-perso! Do you know how to fix the issue when you run pytest from a command line?

[bestekov]

@benoitLebreton-perso , did you manage to have two versions of pyspark installed? Or did you go the route of uninstalling databricks-connect?

The big issue seems to be that installing databricks connect uninstalls the rest of the full pyspark which is extremely annoying. Would be much better to sideload and patch commands only when commands are invoked in a databricks-connect context.

[benoitLebreton-perso]

@benoitLebreton-perso , did you manage to have two versions of pyspark installed? Or did you go the route of uninstalling databricks-connect?

The big issue seems to be that installing databricks connect uninstalls the rest of the full pyspark which is extremely annoying. Would be much better to sideload and patch commands only when commands are invoked in a databricks-connect context.

I uninstalled databricks-connect. I work in local environnement with a local pyspark session. I work with databricks spark session only on notebooks now and I sync my local code with my repos