2D array with missing values or ragged arrays?

[selipot]

I am working towards releasing on AWS a dataset of numerical particles released with Parcels in a global HYCOM simulation. However, my particles can become grounded (or beached) and I am devising a method to flag when this happens. I am debating how to publish the data, as regular 2D arrays with dimensions (trajectory,obs) with an additional flag array to indicate beaching, or as ragged arrays following the clouddrift model? The latest data structure is the basis of the analysis tools of the clouddrift library.

[erikvansebille]

Hi @selipot, this summer I've looked into Parquet as a file-format for parcels: #1399, see also this discussion.
It turned out that Parquet had very poor performance during execution, because appending new data to a Parquet file is slow.

But some people (e.g. @oj-tooth, who develops https://github.com/oj-tooth/lt_toolbox) convert the Parcels zarr-output to parquet before post-processing, after which it can be much faster to work with the data.

So it depends what you want: most parcels users will be used to zarr-output, so if you want a user-friendly (but slightly larger) output I'd go for the regular 2D arrays. Otherwise, ragged arrays or parquet file will be more efficient

[selipot]

Hi @erikvansebille thank you for your reply. I can see how parquet is perhaps optimized for column-oriented data but I tend to see file formats (parquet, zarr, netcdf) as being different from the data structure itself (2D array vs ragged array). As an example the NOAA GDP is now distributed as ragged arrays in a single file but we provide several formats for this file (parquet, zarr, netcdf). So zarr can easily encode ragged arrays too.

So in this case it is less about what I want but what potential users would want. Hence the pool above.

[JamiePringle]

I use ragged arrays in zarr and xarray for using parcels data converted into connectivity matrices, for efficiency reasons. They work, but I will note that I have had a fair number of issues with them because they are less well tested. I expect that would be getting better, as more people like me file bug reports. But just wanted to put that our there.

Examples. In zarr, "zarrVar[index]=ragged_array[indx]" will tend to fail. HDF has issues with figuring out how to deal with the appropriate filter when reading zarr files. All of this can be worked around, but it is still a little rough, and my code has a fair bit of working around edge cases.

For straight parcels output, I prefer the 2D arrays because I can do things like [some_trajectories,all_times] very easily. It does not add much to the file size because Nan's will compress very efficiently when common in the data. Similarly, large blocks of boolean data compress extremely efficiently.

For real world drifter data, I go with ragged arrays all the time.

For examples of my use of ragged arrays, and making them available to R users (for I work with biologists, and this product is oriented towards them): https://github.com/JamiePringle/EZfate

Jamie

[milancurcic]

@selipot and I have discussed this on a call and share the opinion that a regular 2-d array may be more generally useful for reasons @JamiePringle pointed out. We are also considering a ragged-array structure so that CloudDrift and this dataset can piggy-back on each other (i.e. CloudDrift users discover the dataset through the library; dataset consumers discover CloudDrift as the recommended library to load the data).