How to upload a csv in a zip file to /api/v1/dataset/import

[wakandan]

The document is unclear on how to import csv file into /api/v1/dataset/import.

From the test code, I can see that we need some metadata in the zip file to specify which database & dataset to upload the csv file, however there's no guiding on how to add the csv in to the zip bundle.

I tried adding a csv file into the zip file, and then set data=file:///data.csv in the imported_dataset.yaml but it says that it's not a valid URL. What is the correct way to refer to the csv file included in the zip bundle.

And what is the passwords field in the params in the /api/v1/dataset/import for? Let's say I upload the csv file to a public location with password protected, is the passwords field for that purpose?

[wakandan]

hi @betodealmeida can you support this issue? Thanks.

[betodealmeida]

Hi, @wakandan. To upload a CSV you need to add a data attribute pointing to a URL. See this example: https://github.com/apache/superset/blob/master/superset/examples/configs/datasets/examples/users.yaml

The passwords field is used for sending the password for databases, since it's not stored in the export. If you're importing a dataset that points to an existing database you won't need this. But if you're importing a dataset together with a new database that requires a password, you need to either pass passwords in the POST request:

{"databases/mydatabase.yaml": "XXX"}

Or add an attribute password to the file databases/mydatabase.yaml directly.

[Asheboy]

@MPalix86 do you still have the code for this published?

[ericdrosas87]

@Asheboy I have it forked off here as I found it:

https://github.com/ericdrosas87/programmatically_upload_csv_in_superset

As well as my work in progress of running the above code in a Google Cloud Function where the endpoint accepts a JSON payload and parses it out and stuffs it into the Superset DB:

https://github.com/lsst-epo/superset-sideloader

I'm open to collaborating on it as well.

[MPalix86]

i'm actually fixing some issue, is private on monday i will make it public

[LeoDiep]

Hi @MPalix86 ,
I am running Superset v2.1 via docker and when I was trying to use your code, it said:

Error while connecting to the database: connection to server at "13.xxx.xx.xx", port 5432 failed: Connection timed out (0x0000274C/10060)
        Is the server running on that host and accepting TCP/IP connections?

Traceback (most recent call last):
  File "C:\...\programmatically_upload_csv_in_superset\superset_csv_upload.py", line 42, in upload_csv_in_superset
    target_db_info = superset_db.get_data_as_dict_norb(query);
  File "C:\...\programmatically_upload_csv_in_superset\db_conn.py", line 125, in get_data_as_dict_norb
    cursor =self.conn.cursor()
AttributeError: 'NoneType' object has no attribute 'cursor'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\...\programmatically_upload_csv_in_superset\superset_csv_upload.py", line 109, in <module>
    upload_csv_in_superset();
  File "C:\...\programmatically_upload_csv_in_superset\superset_csv_upload.py", line 101, in upload_csv_in_superset
    if(superset_db != target_db):
UnboundLocalError: local variable 'target_db' referenced before assignment

Is it because the docker prevent me to access postgre through TCP/IP connection? Please reccomend what should I change, thank you a lot.

[MPalix86]

Suppose so ! I am using Docker as well, but you should have a separate instance of PostgreSQL that exposes the port specified in the configuration file. I recently updated the script; in the next few days, I will upload the new version. But you should fix the postgres issue with the new version too