Skip to content

Commit 4acd2e4

Browse files
[SDP] spark-pipeline.y(a)ml is the default spec
1 parent 47bfbcb commit 4acd2e4

File tree

1 file changed

+11
-6
lines changed

1 file changed

+11
-6
lines changed

docs/declarative-pipelines/index.md

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,12 @@ Once described, a pipeline can be [started](PipelineExecution.md#runPipeline) (o
4444

4545
## Pipeline Specification File
4646

47-
The heart of a Declarative Pipelines project is a **pipeline specification file** (in YAML format).
47+
A Declarative Pipelines project is defined using a **pipeline specification file** (in YAML format).
48+
49+
Unless specified using spark-pipelines CLI's [--spec](SparkPipelines.md#run) option, Declarative Pipelines uses the following file names as the defaults:
50+
51+
* `spark-pipeline.yml`
52+
* `spark-pipeline.yaml`
4853

4954
In the pipeline specification file, Declarative Pipelines developers specify files (`libraries`) with tables, views and flows (transformations) definitions in [Python](#python) and [SQL](#sql). A SDP project can use both languages simultaneously.
5055

@@ -67,7 +72,7 @@ The following fields are supported:
6772
name: hello-spark-pipelines
6873
catalog: default_catalog
6974
schema: default
70-
storage: storage-root
75+
storage: file:///absolute/path/to/storage/dir
7176
configuration:
7277
spark.key1: value1
7378
libraries:
@@ -486,7 +491,7 @@ $SPARK_HOME/bin/spark-pipelines --help
486491

487492
You've only created an empty Python project so far (using `uv`).
488493

489-
Create a demo double `hello-spark-pipelines` pipelines project with a sample `pipeline.yml` and sample transformations (in Python and in SQL).
494+
Create a demo double `hello-spark-pipelines` pipelines project with a sample `spark-pipeline.yml` and sample transformations (in Python and in SQL).
490495

491496
```shell
492497
$SPARK_HOME/bin/spark-pipelines init --name hello-spark-pipelines && \
@@ -495,7 +500,7 @@ rm -rf hello-spark-pipelines
495500
```
496501

497502
```shell
498-
cat pipeline.yml
503+
cat spark-pipeline.yml
499504
```
500505

501506
=== "Output"
@@ -548,7 +553,7 @@ $SPARK_HOME/bin/spark-pipelines dry-run
548553
=== "Output"
549554

550555
```text
551-
Loading pipeline spec from /Users/jacek/sandbox/hello-spark-pipelines/pipeline.yml...
556+
Loading pipeline spec from /Users/jacek/sandbox/hello-spark-pipelines/spark-pipeline.yml...
552557
Creating Spark session...
553558
Creating dataflow graph...
554559
Registering graph elements...
@@ -571,7 +576,7 @@ $SPARK_HOME/bin/spark-pipelines run
571576
=== "Output"
572577

573578
```text
574-
Loading pipeline spec from /Users/jacek/sandbox/hello-spark-pipelines/pipeline.yml...
579+
Loading pipeline spec from /Users/jacek/sandbox/hello-spark-pipelines/spark-pipeline.yml...
575580
Creating Spark session...
576581
Creating dataflow graph...
577582
Registering graph elements...

0 commit comments

Comments
 (0)