New Task: Release v1.0 of ATN DAC netCDF **trajectory** template

[MathewBiddle]

Who is requesting this?

@ioos/marine-life and NCEI

What is being requested?

ATN establishes a netCDF template for trajectory data. This includes:

• Draft the netCDF template following the contributing guidelines published from Add contributing guidelines ioos-atn-data#79
• Present to IOOS DMAC for review and approval.
• Present netCDF template to NCEI for approval (maybe needs NCEI Dataset Readiness Review (DSRR) board approval?)

What is the requested deadline and why?

2024-12-31

What is the current status quo (i.e., what happens if this does not get done)?

NCEI will not archive data that aren't fully documented. The telemetry community will splinter into multiple file templates for similar observing methods.

What indicates this is done (i.e., how do we know this is complete)?

• A successfully merged pull request to the jekyll base webpage at https://ioos.github.io/ioos-atn-data/ to document v1.0 iteration of the profile.

Provide a description or any other important information.

xref:

• For raw ATN **trajectory** data sets, finalize and implement automated NCEI data archive pathway #41

[mmckinzie]

Considering splitting into two separate issues? – satellite telemetry templates: trajectory and profile or possibly create overarching standardized NCEI netCDF submission milestone or similar, as there will eventually be additional satellite telemetry file types beyond the location (trajectory) and the CTD-F/CTD/TD (profile) files that will need to get formalized as well as generation of templates/files for other telemetry types, e.g., acoustic, multisensor, archival, which will eventually need to get linked to the NCEI ATN collections.

NOTE: Multisensor data effort, focused on DTags is included under a supplemental ML SOW – hope to resume work/effort on integrating these data into the ATN DAC by FY25.

What is being requested?

(ML) ATN DAC trajectory template:

• Develop template - done
• Ensure template is approved by NCEI Dataset Readiness Review (DSRR) board - done
• Publish finalized template and gold standard example and related documentation on ioos website, integrate into ioos compliance checker, link to atn data github – mostly done, needs some revision

(ML) ATN DAC profile template:

• Develop template - in progress
• Ensure template is approved by NCEI Dataset Readiness Review (DSRR) board
• Publish finalized template and gold standard example and related documentation on ioos website, integrate into ioos compliance checker, link to atn data github

(Axiom) Operationalize and automate pipeline from ATN DAC to NCEI WAF, production folder – for both trajectory and profile (finalize trajectory first):

• Successful generation of standardized satellite telemetry trajectory netCDF file - in progress
• Successful generation of standardized satellite telemetry profile netCDF file
• Ability to create these files for all auto-ingested satellite telemetry data from Argos based platforms (w/ or w/o Fastloc or equivalent GPS quality positions) regardless of tag manufacturer and/or tag type
• Automate file generation, from ADR to WAF, push newly flagged deployments to WAF production folder daily
• Integrate ability to list/track NCEI accession numbers in the ADR and mechanism for ATN DC to trigger file generation/release to WAF - done

(Axiom) other related tasks:

• Develop ability to convert manually uploaded data to appropriate netCDF template, link file to ADR
• Link archived files to project’s dataset pages in the ATN portal, embed access link to NCEI using DOI/accession number(s)
• Release/publish and link code used to covert auto-ingested raw satellite telemetry data into ATNs standardized netCDF profiles and/or trajectory file

(ML) other related tasks:

• Finalize ATN collection-level metadata record – see related issue ticket [New Task]: Create collection metadata guidance for ATN #4

Deadlines: finalize satellite telemetry trajectory template, update IOOS website content and ensure files are successfully generated by Axiom and able to be pushed to WAF production folder – July 1, 2024; Operationalize/Automate NCEI pipeline for trajectory files – September 1, 2024; Release/publish trajectory template code – September 1, 2024; Link NCEI files to ATN portal – November 30, 2024; Profile template development and IOOS website documentation – November 30, 2024; Successfully generate profiles files and serve to NCEI – add to YR3 ML SOW

[MathewBiddle]

@mmckinzie I agree this should be split into two issues. Thank you for the additional details.

I think some of your comments can stay here and some can be moved to the pipeline development tickets that have been xref'ed above.

From your comments it sounds like we need to adjust this ticket and create a new one so we have two tracking mechanisms (documentation of profile and trajectory templates). The thinking is these tickets are strictly for documenting and publishing the netCDF template. Not necessarily for developing the archival pipeline or cyberinfrastructure to implement said pipeline. So, I propose we adjust the title of this issue and create a second issue as follows:

• Publish ATN DAC trajectory netCDF template
• Publish ATN DAC profile netCDF template

Both of those will be intrinsically linked to the "develop the archive pipeline" in #41 and #42. The first step in those two issues is to define the netCDF structure, so we can make this piece a dependency.

As new formats/pipelines come up (e.g., acoustic, multisensor, archival) we will create issues to manage the development of the template and the establishment of a pipeline to NCEI. For now, let's focus on the trajectory and profile parts.

Do you mind creating a new issue for this?

NOTE: Multisensor data effort, focused on DTags is included under a supplemental ML SOW – hope to resume work/effort on integrating these data into the ATN DAC by FY25.

The issue should be concise and explicitly about the task at hand. @laurabrenskelle wrote a contributing guide to help frame the scope of a new issue. If you have questions don't hesitate to reach out!

[MathewBiddle]

Title

Publish ATN DAC trajectory netCDF template

Who is requesting this?

@ioos/marine-life

What is being requested?

ATN DAC trajectory and profile standardized templates/NCEI netCDF file format documentation and code

• Draft the netCDF template following the contributing guidelines published from Add contributing guidelines ioos-atn-data#79
• Present netCDF template to NCEI for approval (maybe needs NCEI Dataset Readiness Review (DSRR) board approval?)

What is the requested deadline and why?

2024-07-01

What is the current status quo (i.e., what happens if this does not get done)?

NCEI will not archive data that aren't fully documented. The satellite telemetry community will splinter into multiple file templates for similar observing methods.

What indicates this is done?

• A successfully merged pull request to the jekyll base webpage at https://ioos.github.io/ioos-atn-data/ to document the current standard.

Provide a description or any other important information.

[MathewBiddle]

I am updating the description of this issue to adjust:

• That this topic should be two issues. One for the trajectory template and one for profile template.
• Clearly articulate what is being done.
• This issue will now be for tracking progress on the trajectory template development.

[MathewBiddle]

@mmckinzie @laurabrenskelle I'm wondering if it would be cleaner if we closed this issue and opened a new one without all the changes?

[MathewBiddle]

Website needs to be updated to align with changes to the template recently.

Website should be updated to reference to a production file.

@mmckinzie to send final file once complete

[MathewBiddle]

Curious if it would be worthwhile to put a small paper together describing the netCDF template? There is a similar paper for camera trap data at https://zslpublications.onlinelibrary.wiley.com/doi/10.1002/rse2.374, for reference.

The also published CameraTrapDP on tdwg (https://camtrap-dp.tdwg.org/) and I'm wondering if we could make https://ioos.github.io/ioos-atn-data/atn-sat-telem-specification-v1-0.html a data exchange format for satellite telemetry data. We've done most of the work, now it would be formalizing that within tdwg.

[mmckinzie]

I am still working on resolving a few lingering issues with the trajectory template, successful file generation and metadata documentation with Axiom and NCEI. Once resolved, I'll get the trajectory template finalized and updates pushed to the specification website. Once complete, I do think it would be worthwhile to generate a short descriptive paper about our template(s). It would provide a nice complement to this effort. I'm also open to the idea of creating a data exchange format for this template, but not really sure what that would entail.

[mmckinzie]

I think we need to create another related task for this effort, see comments from Apr 28, in regard to operationalizing and automating the pipeline from ATN DAC to NCEI WAF. This is presently a manual process. It will require additional development work by Axiom to fully automate this workflow and ensure new files are being successfully generated after NCEI flags embedded within the ADR and push to the WAF production folder for harvesting by NCEI.

New issue ticket appears to have already been created and tasking is captured in issue #41

[MathewBiddle]

satellite trajectory template can be found at https://github.com/ioos/ioos-atn-data/blob/gh-pages/_docs/data/templates/atn_satellite_trajectory_template.cdl

This is the template that Axiom should be working from to generate the files needed to complete #41.

@mmckinzie I've added comments to the cdl representation to document what should go in some of the variables/attributes and where the guidance comes from. I will be using that cdl template to update the website https://ioos.github.io/ioos-atn-data/atn-sat-telem-specification-v1-0.html soon. I'll post here once I've put together a pull request.

[mmckinzie]

@MathewBiddle I only quickly looked at the template linked above but I noticed a few things in the cdl we should discuss before the website is updated. For example, you have :attachment listed as a global attribute. This is not part of our template; this is one of the misc attributes that occasionally gets carried over from the WC data files. We do not collect this information in the ADR. I also spotted a few things within the variables that seemed off related to reported fill values and the QARTOD flags. Anyways, please do not update website until we have a chance to discuss.

[mmckinzie]

@MathewBiddle cdl file should be based off of what is in this 'template' file.
Here is another example.

Again, please do not share these google docs publicly as they contain embargoed information within the 'example' columns.

[mmckinzie]

Also, just as a reminder the current specification and cdl is only applicable to Wildlife Computer data files from select tag types.

The global attributes will be the same regardless of the manufacturer/tag type, but the variables will be different. I'm working on updating the google doc template to include a relevant SMRU example and list of variables.

[MathewBiddle]

Also, just as a reminder the current specification and cdl is only applicable to Wildlife Computer data files from select tag types.

We need to discuss this more. Why do other manufacturers and tag types need another spec?

What we are developing should be broad enough to provide the capacity to encapsulate nuances between manufacturers within the same metadata attributes. Obviously the content will be different, but the attributes will be the same, thus no need for another spec. it's just another implementation of said spec.

[mmckinzie]

@MathewBiddle some of the other things I wanted to discuss relate to the :publisher_email and the :reference link listed in association with the QARTOD variables. We presently have [email protected] listed as the publisher email, but I am not sure who manages or has access to this account, so just wanted to verify this is still the best email address to list in association with that global attribute. Then we have https://cdn.ioos.noaa.gov/media/2020/03/QARTOD_TS_Manual_Update2_200324_final.pdf listed as the reference for the QARTOD variables, as these are not profile data and do not contain salinity and/or temperature profile data, it doesn't seem like this is the most appropriate reference. Would it be better to list the Argo manual instead? https://cdn.ioos.noaa.gov/media/2020/03/Argo-QC-for-CTD-and-Trajectory-Data.pdf or possibly this file: https://cdn.ioos.noaa.gov/media/2020/07/QARTOD-Data-Flags-Manual_version1.2final.pdf

[MathewBiddle]

Both of these are great questions. I wouldn't stress to much about having them perfect right now. What's in there is fine for v1.0.

For the :references attributes, it would probably be best to point to the Ocean Best Practices DOI for the appropriate manual. For example, http://dx.doi.org/10.25607/OBP-878 for the Control Flags manual. But, again, that can be updated in a future version. You can find all the manuals in OBPS at https://repository.oceanbestpractices.org/handle/11329/336

[mmckinzie]

I agree the original intent was to develop a specification that was general or board enough to be applicable to all manufacturers. But in the end that was not what happened. I have brought up this issue numerous times in the past and agree it is worthy of further discussion.

The data files the DAC auto-ingests from the various manufacturers are all different even from analogous tag types, they contain different suite of variables so presently the NetCDF files also contain a different suite of variables depending on manufacturer. There are of course variables that are common to all data files regardless of manufacturer but then again, a suite of variables that are unique to each manufacturer. See here.

This issue is applicable to both the trajectory and profile components of these data.

[mmckinzie]

remove :attachment and :tag_serial from global attributes in cdl

See here.

[mmckinzie]

We may also want to consider updating what is listed in the :acknowledgment attribute when generating a more board or generic version of this template that is not manufacturer specific.

[MathewBiddle]

This is done. See the v1.0 specification at https://ioos.github.io/ioos-atn-data/atn-sat-telem-specification-v1-0.html

[MathewBiddle]

This is complete through:

• adding trajectory template ioos-atn-data#88
• updating template ioos-atn-data#90

Moving the issue to Done as we've completed the initial task