Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to import/export data to/from Parquet files with SQLCMD #486

Open
dlevy-msft opened this issue Dec 7, 2023 · 1 comment
Open
Labels
enhancement New feature or request
Milestone

Comments

@dlevy-msft
Copy link

Creating this placeholder for the investigation of adding import/export functionality to go-sqlcmd.

Main use cases would include:

  1. Output SQL Server query results to parquet files
  2. Read data from parquet files for display of some or all rows on screen to check contents of file
  3. Read data from parquet files and insert into table in SQL Server database
  4. Be able to read both standalone parquet files as well as delta
  5. Perform basic filtering via SQL-like where clause or regex by column

Additional possibilities include:

  1. Output SQL Server query results to json, csv, xls files
  2. Convert json, csv, xls to parquet
  3. Convert parquet, including delta, to json, csv, xls
  4. Convert subset of rows between formats - ex: "Take customer data from delta store and export all customers in Chicago to xls"
@dlevy-msft dlevy-msft added the enhancement New feature or request label Dec 7, 2023
@dlevy-msft dlevy-msft added this to the Backlog milestone Dec 7, 2023
@flarco
Copy link

flarco commented Jul 2, 2024

You can do this easily with sling, check out: https://github.com/slingdata-io/sling-cli

# set connection via env var
export mssql='sqlserver://...'

# test connection
sling conns test mssql

# run export for many tables
sling run --src-conn mssql --src-stream 'my_schema.*' --tgt-object 'file://{stream_schema}/{stream_table}.parquet'

# run export for one table
sling run --src-conn mssql --src-stream 'my_schema.my_table' --tgt-object 'file://my_folder/my_table.parquet'

# run export for custom sql
sling run --src-conn mssql --src-stream 'select col1, col2 from my_schema.my_table where col3 > 0' --tgt-object 'file://my_folder/my_table.parquet'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants