Shell script to grab formatted variables to copy and paste into config yaml file

[justin-richling]

Adding variables to the config yaml files seems like it could be daunting if many variables are needed and especially if the set of variables change from run to run.

I have built a shell script to automatically grab all float variables (float for now, can adjust later) from one of the h0 netCDF files associated with a run. The script creates a text file that has formatted variables with spaces and - to match how they are in the yaml file.

Once the script runs, one could open the text file, copy the contents and just paste into the config yaml file under the variables section.

The script is located here: /glade/work/richling/ADF/ADF/ncdump_float_var.sh

And the only argument needed to run with the script is location to dataset that has the h0 files, example:
ncpath=/glade/scratch/hannay/archive/f.c6_3_41.FWscHIST.ne30_L58_macmic6.001/atm/hist/
./ncdump_float_var.sh $ncpath

Sample output text file located here: /glade/work/richling/ADF/ADF/f.c6_3_41.FWscHIST.ne30_L58_macmic6.001.cam.h0.1989-12_var_list.txt

   - ACTNL
   - ACTREI
   - ACTREL
   - AODDUST
   - AODVIS
   - BURDENBC
   - BURDENDUST
   - BURDENPOM
   - BURDENSEASALT
   - BURDENSO4
   - BURDENSOA
   - CAPE
      .
      .
      .

I'd be happy to integrate this to the project, but wanted a discussion about if this would be desirable (do you use a better method when populating your yaml files?), if the script works for everybody's configuration, where should the script live if it is accepted, or any other thoughts, comments, or concerns.

Let me know what you think!

[andrewgettelman]

1. I don't think we should have any shell scripts in the ADF. If we add this to the ADF, it should probably be in python. I guess I am fine with having this, but not sure we need to add it to the ADF.
2. Is the purpose just to automatically pull out variable names from a file? I think the ADF should be a bit more selective than that: it doesn't tell you whether variables are 2-D or 3-D for example.

[justin-richling]

Thanks for the input on not including a shell script, I think I understand why this is probably not preferred and could be redone in python.

The idea behind it was to automatically pull variable names out of h0 files and format them in a way to simply copy and paste into the diag_var_list in the configuration yaml file that runs with run_adf_diag rather than populate the list manually if that makes sense.

I've only worked a couple of examples with running the ADF, and I don't know if I've dealt with 3-D variables. In the diag_var_list do you designate variables there based on number of dimensions? I'm still trying to wrap my head around how all the pieces work together.

[cecilehannay]

I agree we don't want to include in the ADF. But Justin's script is useful until we have another way to do that.

In the long term, we could have a flag requesting the ADF to plot all the variables in the h0.
It could be something like:
plot_all_ho_var : true => we plot every variables in the h0 file
plot_all_ho_var : false => we plot the variables from diag_var_list

But until this is ready, Justin's script is useful.

At the last two hackathons, we talked about creating a new repo for the scripts/Jupyter Notebooks provided by the community. This is where this kind of scripts could live. Same for the ADF asset browser from Dan.
This repo would provide scripts as is while everything on the ADF repo is vetted.

@nusbaume was going to create a new repo that we could start push some scripts/notebooks. We need to decide on a name. Any suggestions?

I am thinking maybe:
https://github.com/NCAR/ADF_community