Dataform tools - a vscode extension

⚠️ This is not an officially supported Google product.

VS Code extension for Dataform with following features

Feature	Description
Dependancy graph	Interative dependancy graph with external sources higlighted in distinct colors
Inline diagnostics on .sqlx file ❗	Native lsp like experience with diagnostics being directly put on both the sqlx file & compiled query
Go to definition	Go to definition for source in $ref{("MY_SOURCE")}. Takes you to MY_SOURCE.sqlx or sources.js at the line where MY_SOURCE is defined
Auto-completion	- declarations in ${ref("..")} trigger when $ character is typed - Dependencies when " or ' is typed inside the config block which has dependencies keyword is in the line prefix - tags when " or ' is typed inside the config block which has tags keyword is in the line prefix
Code actions	Apply dry run suggestions at the speed of thought
Compilation & Dry run stats	- Live compiled query in a vertical split on save which is in sync with the current cursor position of your .sqlx file - Data processed by query on bottom right on successful dry run
Run a specific file/tag	Run a file/tag, optionally with dependencies/dependents with vscode command pallet / menu icons
Format using Sqlfluff 🪄	Fromat .sqlx files using sqlfluff

Requirements

1. Dataform cli

   npm i -g @dataform/cli

2. Setup default application credentials for GCP

3. To enable formatting using sqlfluff install sqlfluff

   # install python and run
   pip install sqlfluff

4. To enable prettier diagnostics install Error Lens extension [ optional ]

Note

Trouble installing ? Please see FAQ section, if you are still stuck, please raise an issue here

Features

Dependency graph

Inline diagnostics errors on .sqlx files ❗

Autocomplete model, tags, dependencies

Auto completion support for dependencies when " or ' is typed inside the config block which has dependencies keyword is in the line prefix

Declarations in ${ref("..")} trigger when $ character is typed

Auto completion support for tags when " or ' is typed inside the config block which has tags keyword is in the line prefix

Go to definition

Go to definition for source in $ref{("MY_SOURCE")}. Takes you to MY_SOURCE.sqlx or sources.js at the line where MY_SOURCE is defined

Compilation & Dry run stats

Live compiled query in a vertical split on save which is in sync with the current cursor position of your sqlx file. Data processed by query on bottom right on successful dry run

Formatting using sqlfluff

Run file/tag with dependencies/dependents

Open vscode command pallet by pressing CTLR + SHIFT + p or CMD + SHIFT + p on mac and run one of the required commands

Commands
Dataform: Run current file
Dataform: Run current file with dependencies
Dataform: Run current file with dependents
Dataform: Run current tag
Dataform: Run current tag with dependencies
Dataform: Run current tag with dependents
Dataform: Format current file

Frequently asked questions

1. Unable to execute command e.g. error command vscode-dataform-tools.xxx not found

   • It is likely that the vscode workspace folder is not opened at the root of your dataform project. For example, if your dataform project is located at ~/Documents/repos/my_dataform_project ensure that workspace is opened at ~/Documents/repos/my_dataform_project NOT ~/Documents/repos/my_dataform_project
   • The above design is to facilitate the exection of dataform compile --json command without infering the dataform root at run time

2. Error compiling Dataform, process existed with exit code 1

   • Check if correct dataform cli version is installed by running dataform --version in your terminal
   • Ensure that dataform cli version matches the version required by the project
   • Try compiling the project by running dataform compile on your terminal from the root of your dataform project
   • In case you need to install a specific dataform cli version by running npm i -g @dataform/[email protected]. Make sure you verify the version by running the dataform --version
   • In case the error is not due to all the above reasons it is likely that you have a compilation error in your pipeline

3. Dataform encountered an error: Missing credentials JSON file; not found at path <your_project_path>/.df-credentials.json

   • Run dataform init-creds from the from the root of your dataform project in your terminal
   • You will be promted to pick the location and type of authentication json/adc. Choosing adc will be use your default gcp credentials that you had setup using gcloud

4. I do not want to see compiled query each time I save it

   • Open vscode settings and search for Dataform and uncheck the following setting

5. I want the autocompletion to be of the format ${ref('dataset_name', 'table_name)} instead of ${ref('table_name')}

   • Open vscode settings and search for Dataform and select the prefered autocompletion format

Known Issues

• Features such as go to definition / dependancy graph might not work with consistantly with ${ref("dataset", "table")}
• sync feature flickers when user tries to scroll a non-active editor. Fixes when user selects the active editor by clicking on it

TODO

• Preview query results in a table
• Bundle javascript files in the extension using esbuild or webpack
• Handle case where user is not connected to internet or on vpn where network request for dry run cannot be made
• Add proper logging, winston-transport-vscode