How to export file from csv -> hdf5 without generating yaml file. ?

[Quy-Dev02]

I use a library to save CSV files as hdf5. However, when creating hdf5 files, YAML files are also created. I have researched the document but can't figure out the effect of this file. Is there any option so that when exporting to an hdf5 file, it will not automatically generate YAML file?

[JovanVeljanoski]

Hi,

Do you understand the point of that file? In a sense.. that file contains metadata about the conversion process. So you can do something like

df = vaex.open('file.csv', convert=True) 

The first time you run the line above, the CSV file will be converted to HDF5. The 2nd time you run it, Vaex will just read the HDF5 file directly, and knows exactly what to read thanks to that YAML file. YAML file also includes some metadata of the source file (the CSV) and the converted file (the HDF5). So if for some reason the source or the output file is modified (like due to an update or whatever), running the line above will re-trigger the conversion.

If you do not want this, you can simply delete the yaml file. If you do not want it created, then convert data yourself like this

df = vaex.open('file.csv', convert=True) 
df.export('file.hdf5')

I don't think the 2nd approach adds that much complexity :). In the past, using convert=True in open was maybe more practical if the CSV is huge and can not fit it memory, but now vaex can stream over CSV data (i.e. use it in an out of core manner), exporting to HDF5, parquet etc.. is trivial.

Does this help a bit?

[Quy-Dev02]

Hi, @JovanVeljanoski
Thanks for answering. it helped me understand the reason why there is a yaml file. Thanks so much for this answer.