when ADF finds multiple TS files, should it choose one or quit with an error?

[bitterbark]

Currently if there are multiple time-series files, the ADF quits (at least in this call).
AdfError: Currently the AMWG table script can only handle one time series file per variable. Multiple files were found for the variable 'Q'

First, I would like it to give a list of the files it found, so I don't have to dig back in my config file to find the path. But more importantly: should it choose the one it thinks is mostly likely to be right, and just generate a warning, or should it fail (current behavior).

[bitterbark]

I see that this is being treated differently by different scripts.

In create_climo_files.py:process_variable() and averaging_example.py

#Read in files via xarray (xr):
if len(ts_files) == 1:
cam_ts_data = xr.open_dataset(ts_files[0], decode_times=True)
else:
cam_ts_data = xr.open_mfdataset(ts_files, decode_times=True, combine='by_coords')

In amwg_table.py

if len(ts_files) != 1:
errmsg = "HELLO! Currently the AMWG table script can only handle one time series file per variable."
errmsg += " Multiple files were found for the variable '{}'".format(var)
raise AdfError(errmsg)

Note that the former would cause problems in my case: I have timeseries files from the same model case but with different naming conventions due to different versions of the ADF. Note that this could be resolved by overwriting the output directory, but if we have a setting like that we'll have to be careful to not lose existing files!

Furthermore, there is an inconsistency in how the two functions search for the file names:
create_climo_files.py:process_variable() and averaging_example.py

      ts_filenames = search.format(CASE=case_name, VARIABLE=var)
      ts_files = sorted(list(input_location.glob(ts_filenames)))

amwg_table.py

        ts_filenames = '{}.*.{}.*nc'.format(case_name, var)
        ts_files = sorted(list(input_location.glob(ts_filenames)))

Maybe there is a reason they need to be different, and so far I think they both work but could it be time to write a function to do this that every script can call? Given the checks (if not ts_files, len(ts_files)) it would cut a nice chunk of code from each script too.

[brianpm]

There are several issues here.

First: different methods for retrieving the time series
I agree that we should probably have a standard way to get the time series files that all the scripts could use.

Second: Inconsistent searching methods
The two searches (create_climo_File vs amwg_table) are actually functionally the same. In create_climo_file, I made search a keyword argument as a workaround to potentially using non-ADF file naming conventions. Maybe something like this could be used in the general utility function. This comes up in many potential cases when the time series files are made outside of ADF.

Third: finding multiple time series files
As we have recently discussed at the hackathon, it will be a common issue that time series are split into multiple files. For example, analysis of the CESM2 CMIP6 historical simulation(s), where some of the ensemble members are split into multiple files while others are one single file (e.g., ls /glade/campaign/collections/cmip/CMIP6/timeseries-cmip6/b.e21.BHIST.f09_g17.CMIP6-historical.*/atm/proc/tseries/month_1/*.T.*.nc)

For this kind of anticipated scenario, I think using open_mfdataset is the most appropriate solution.

Fourth: different time series files from the same case in the same directory
I'm not sure what to do about this scenario. I could imagine other versions of this happening. Examples: (1) running ADF on different time slices of the same simulation, (2) running ADF incrementally while a simulation is running. In the 2nd case, maybe the files would be or should be overwritten. The complication is that as long as ADF is directly dealing with files, handling these edge cases seems like it would come down to the user being forced to provide specific search strings or regular expressions. If we eventually move to a catalog-based approach, maybe some of these complications would go away, but that might depend on how we set up the catalog. Anyway, I'm stumped on this specific issue.