Data archive through February 28, 2025

[MathewBiddle]

Who is requesting this?

Marine Life

What is being requested?

• Employ workflow currently in-place (manual) to facilitate netCDF package delivery of trajectory datasets (format of which has already been approved by NCEI and finalized) for archive at NCEI
• Prioritized list of deployments will be flagged by ATN DAC Data Coordinator

What is the requested deadline and why?

2025-02-28

What is the current status quo (i.e., what happens if this does not get done)?

• data not making it to NCEI thus we are not meeting our PARR requirements.

What indicates this is done (i.e., how do we know this is complete)?

• 300 deployments archived at NCEI

Provide a description or any other important information.

No response

[MathewBiddle]

Moved to In progress as we should be working on this now.

[iwensu0313]

Update: no new datasets to archive to NCEI yet since Dec 1, 2024.

[MathewBiddle]

Any updates on this? I haven't seen any notifications from NCEI that new data have been archived.

[iwensu0313]

@mmckinzie hasn't mentioned any new datasets for archival to NCEI (checked the ATN data meeting notes from last week when I was at the conference), so no updates to my knowledge

[iwensu0313]

Oh wait it's automatically flowing in staging. Here's the staging link that shows what's flowing: https://stage-ncei-waf.srv.axds.co/atn/test/ (see related #41). The last thing I see in there is from 20-Nov-2024. Does that check out?

Was waiting for feedback or the green light from IOOS Marine Life team I think to move staging to production.

[mmckinzie]

@iwensu0313 Do NOT move any of the files from the test directory to the production folder. They are not ready to go. They do not have the correct list of contributors/creators. And some of the NCEI flags in the ADR were not embedded by me.

[mmckinzie]

I will follow-up later today with the list of deployments that are ready to move to production.

[MathewBiddle]

@mmckinzie per your last update I was expecting more deployments to be pushed to NCEI. I haven't seen those come through. Anything we can do to advance this?

[mmckinzie]

@MathewBiddle An issue with the automated NCEI pipeline/archiver tool in the ADR was discovered that was causing problems with file generation for deployments with reused ptt ids that was either resulting in files to fail or the wrong file to be generated, so I have been unable to push any new files to NCEI. Axiom was looking into it and seems they have found a potential solution, but it requires me to make while workable, not necessarily desirable changes to PIs deployment metadata. I'm working on that now so I can generate another batch of test files to confirm the issue has in fact been resolved. If it has, I can resume flagging deployments for release, in theory.

But, there is also another related issue that was discovered with regards to hooking deployments up to the portal, also potentially related to reused ptt ids, that is causing previously hooked up deployments to be removed from portal if/when a new deployment with the same ptt is hooked up even if the ids are associated with a different projects/species and have different unique deployment ids. This in turn breaks the connection for data/deployments to the ADR for any ids that were unintentionally removed from the portal and if data/deployment are not connected to the portal and data are not linked to the ADR, I cannot autogenerate the NCEI files, which is also preventing me from releasing data. We're still looking into this issue. Both issues need to be resolved before we can reliably generate the NCEI files.

[MathewBiddle]

Thanks @mmckinzie ! Hopefully those issues can be sorted out and we can continue pushing data to NCEI.

@iwensu0313 any idea on LOE for these fixes?

[iwensu0313]

Not sure what LOE stands for but this would likely be a med/high effort as the source of the issue is unknown so would require extensive troubleshooting, planning, then fixing, then testing. Given the complexity of the requirements and connectivities, the best way to approach this would be to make sure we're addressing all concerns and end requirements instead of one-off fixes. Will share my ATN summary doc later today so we can decide how to move forward over this next bit

[mmckinzie]

ADR archiver tool logic improvements along with listing the unique manufacturer/vendor ID in the deployment ID field instead of a unique PI provided ID appears to have resolved the issue with generating NCEI files for deployments with reused ptt IDs.

All 29 files from ADR project 32 are now being generated correctly. OK to move these files from the staging WAF test directory to the prod folder for harvesting by NCEI.

Likely also OK to go ahead and updated the ADR archiver tool pathway to send NCEI files directly to the prod folder instead of the staging WAF test directory.