GSoC: Workflow Overview and Missing Functionalities for Glider QA/QC

[Sakshamgupta90]

Hi @ocefpaf @mwengren @MathewBiddle,
As I am delving deeper into the project, I believe having a clear understanding of the workflow would greatly enhance my productivity and ensure that my contributions align well with the project's goals. Could you please provide a brief overview of the current workflow for the project?
Additionally, it would be very helpful if you could highlight any specific functionalities or checks that are currently missing or need improvement. This will help me to focus my efforts more effectively and contribute in the most meaningful way possible.

[ocefpaf]

@Sakshamgupta90 I pinged you on slack on this topic. We can use the document I shared with you to start narrowing down the test we need to implement and/or modify. We can keep this issue open and start listing them here as we read the doc.

[Sakshamgupta90]

Hi @ocefpaf,
I will make sure to list everything on this PR. And will update you by providing the list of functions that need to implement or modify.

[Sakshamgupta90]

Hi @ocefpaf,
List of All the Tests, that are implemented:

• Test 3) Location Test (Required) -> Link
• Test 4) Gross Range Test (Required) -> Link
• Test 5) Pressure Test (Required) -> Link
• Test 6) Climatological Test (Strongly Recommended) -> Link
• Test 7) Spike Test (Strongly Recommended) -> Link
• Test 8) Rate of Change Test (Strongly Recommended) -> Link
• Test 9) Flat Line Test (Strongly Recommended) -> Link
• Test 11) Attenuated Signal Test (Suggested) -> Link
• Test 14) Density Inversion Test (Suggested) -> Link

I checked the code base, and found that these 5 tests are not yet implemented:

• Test 1) Timing/Gap Test (Required)
• Test 2) Syntax Test (Required)
• Test 10) Multi-Variate Test (Suggested)
• Test 13) TS Curve/Space Test (Suggested)
• Test 12) Previous Profile Test (Suggested)

Out of these 5 Tests, first 2 are from Group1 (i.e. Required Tests) and rest are from Group3 (Suggested Tests).

[ocefpaf]

Timing/Gap and Syntax will require some raw data examples that we request i tomorrow's meeting. The TS one is quite challenging to code and we should start with a simple case for it. The challenges are:

1. Coastal data may differ from classic TS fits.
2. There are two options there, fitted equation or a numeric table.
3. The tests are assuming that the temperature sensor is better than the salinity one, which is usually true, and compare the expected salinity based on temperature alone.

Can you updated your message above to list all tests, checkmark those that already exist and add a link in the code base so we can track them? Some, while may be already coded, may require some adjustments for the glider case and we need to check.

@leilabbb may have some insight for us here on which test is OK and which is not in the existing ones.

[ocefpaf]

@Sakshamgupta90 following up from today's meeting. We should get the notebook and test data from #106 and run all the tests that are already implemented. According to @leilabbb some tests results are not working as expected and we need to fix them.

[leilabbb]

@ocefpaf @Sakshamgupta90 can you put a link to the Slack channel to join the conversation?

[ocefpaf]

Leila I invited you using the slack interface. In any case, see if this link works.

[leilabbb]

These are the tests the Glider DAC is interested in:

To be added to the QC workflow:
Test 3) Location Test (Required) -> Link

Already in the QC workflow:
Test 4) Gross Range Test (Required) -> Link
Test 7) Spike Test (Strongly Recommended) -> Link
Test 8) Rate of Change Test (Strongly Recommended) -> Link
Test 9) Flat Line Test (Strongly Recommended) -> Link

The links below give access to a notebook with the Glider DAC QC workflow, the example netCDF data file used in the notebook and the configuration yaml file needed to run the the QC process.

GLIDER-DAC QC Notebook

Example netCDF File

Configuration Yaml File

IMPORTANT:
In the notebook, the last section checks the flags and you will see the differences I discovered implementing the QARTOD tests in two different ways.

[Sakshamgupta90]

Hi @leilabbb @ocefpaf,
I've observed differing results between the two methods used to implement the QARTOD flat line tests. I'm a bit confused because both methods show different outcomes. Could you please advise on which method you find most appropriate?

These are the tests the Glider DAC is interested in:

To be added to the QC workflow: Test 3) Location Test (Required) -> Link

I've reviewed the location tests for both general models and glider dac, and their descriptions appear quite similar in terms of checking latitude, longitude, and displacement. Could you clarify what specific adaptations are needed for implementing to make sure I'm in sync?

[leilabbb]

METHODE 2: Using the ioos_qc.qartod.flat_line_test is more accurate. It shows more reasonable test results.

For the Location test, it is on the list to add to the Glider DAC QC workflow, so I do not have any comment on that.

[Sakshamgupta90]

For the Location test, it is on the list to add to the Glider DAC QC workflow, so I do not have any comment on that.

You mean the list that I created above? If yes, then I listed Location test in the implemented section.

[Sakshamgupta90]

Hi @ocefpaf,
can you share some raw data so that I can start implementing timing and syntax test.

[ocefpaf]

Aren't we focusing on the bug that @leilabbb described in her notebook? We should tackle 1-thing at a time and see if we can complete some of the goals. I'd like for you to prioritize this and #124.