Skip to content
This repository has been archived by the owner on Jun 14, 2024. It is now read-only.

Change the quick guide with Spark Exetension use cases #512

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 59 additions & 13 deletions docs/_docs/01-ug-quick-start-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -239,16 +239,62 @@ hs.explain(query, verbose = True)

#### Enable Hyperspace

Now that you have created an index that your query can utilize, you can enable Hyperspace and execute your query:

Scala:
```scala
spark.enableHyperspace
query.show
```

Python:
```python
Hyperspace.enable(spark)
query.show()
```
Now that you have created an index that your query can utilize, you can enable Hyperspace and execute your query.
There are two ways to enable Hyperspace:

1. Using Hyperspace extension
paryoja marked this conversation as resolved.
Show resolved Hide resolved

You can start a session with Hyperspace extension.
The follows are examples to use Spark™ with Hyperspace.

spark-submit:
```
spark-submit -c spark.sql.extensions=com.microsoft.hyperspace.HyperspaceSparkSessionExtension ...
```

Scala:
```scala
val spark = SparkSession
.builder()
.appName("...")
.master("...")
.config("spark.sql.extensions", "com.microsoft.hyperspace.HyperspaceSparkSessionExtension")
.getOrCreate()
```

Python:
```python
from pyspark.sql import SparkSession

spark = SparkSession \
.builder \
.appName("...") \
.master("...") \
.config("spark.sql.extensions", "com.microsoft.hyperspace.HyperspaceSparkSessionExtension") \
.getOrCreate()
```

Spark configuration file (default: `$SPARK_HOME/conf/spark-defaults.conf`):

```cs
# Add the following line to the configuration file.

spark.sql.extensions com.microsoft.hyperspace.HyperspaceSparkSessionExtension
```


2. Calling enableHyperspace function explicitly

By explicitly calling enableHyperspace (for Scala) or Hyperspace.enable (for Python), Spark will use Hyperspace indexes when it is applicable.

Scala:
```scala
spark.enableHyperspace
query.show
```

Python:
```python
Hyperspace.enable(spark)
query.show()
```