Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove stream config #100

Merged
merged 3 commits into from
Mar 16, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
121 changes: 0 additions & 121 deletions docs/source/examples/usage/StreamConfig.ipynb

This file was deleted.

33 changes: 2 additions & 31 deletions docs/source/usage.rst
Original file line number Diff line number Diff line change
Expand Up @@ -69,8 +69,7 @@ Configurations

Configuration objects represent a collection of quality control tests to run and the parameters for each one. There are three main types of `Config` objects:

- StreamConfig_: This configures QC tests for a single stream of data like a ``list``, ``tuple``, ``numpy.ndarray``, ``dask.array``, ``pandas.Series``, ``netCDF4.Variable``, or ``xarray.DataArray``. This can be used standalone, or as a building block for the following more complex configs.
- ContextConfig_: This defines a collection of ``StreamConfig`` objects. These can be applied to multiple variables provided in a ``pandas.DataFrame``, ``dask.DataFrame``, ``netCDF4.Dataset``, or ``xarray.Dataset``. Optionally, these configs can be constrained to specific time domains (``windows``) -- and/or spatial domains (``regions``).
- ContextConfig_: This defines a collection of ``ConfigTypes`` objects. These can be applied to multiple variables provided in a ``pandas.DataFrame``, ``dask.DataFrame``, ``netCDF4.Dataset``, or ``xarray.Dataset``. Optionally, these configs can be constrained to specific time domains (``windows``) -- and/or spatial domains (``regions``).
- Config_: A collection of ``ContextConfig`` objects, suitable for configuring a single input dataset to be broken up by region and time window before having QC checks applied.

Each configuration type can be initialized through Python objects or from files and can be represented in the following ways:
Expand All @@ -83,37 +82,9 @@ In addition, the ``ContextConfig`` and ``Config`` objects can be initialized wit
- netCDF4/xarray filepath (``str`` or ``Path`` object) or ``Dataset``


StreamConfig
~~~~~~~~~~~~
A ``StreamConfig`` object defines a specific `ioos_qc` test module and test function along with the configuration parameters in which to run it with.

.. note::

In earlier versions, ``StreamConfig`` was known as ``QcConfig``.

Usage
^^^^^

.. code-block:: python
:linenos:
:caption: A basic ``StreamConfig`` object

from ioos_qc.config import StreamConfig

config = {
'qartod': {
'gross_range_test': {
'suspect_span': [1, 11],
'fail_span': [0, 12],
}
}
}
c = StreamConfig(config)


ContextConfig
~~~~~~~~~~~~~
A ``ContextConfig`` object defines multiple ``StreamConfig`` objects as well as optional `region` and `window` objects.
A ``ContextConfig`` object defines multiple ``ConfigTypes`` objects as well as optional `region` and `window` objects.

region
^^^^^^
Expand Down
10 changes: 5 additions & 5 deletions ioos_qc/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -261,7 +261,7 @@ def __init__(self, source, version=None, default_stream_key='_stream'):
# Return a list with just one ContextConfig
self._calls += list(ContextConfig(self.config).calls)
elif dict_depth(self.config) >= 4:
# This is a StreamConfig
# This is a Config
self._calls += list(ContextConfig(odict(streams=self.config)).calls)
else:
# This is a QcConfig
Expand Down Expand Up @@ -353,7 +353,7 @@ def add(self, source) -> None:


class ContextConfig:
"""A collection of a Region, a TimeWindow and a list of StreamConfig objects
"""A collection of a Region, a TimeWindow and a list of Config objects

Defines a set of quality checks to run against multiple input streams.
This can include a region and a time window to subset any DataStreams by before running checks.
Expand All @@ -366,11 +366,11 @@ class ContextConfig:
ending: 2020-04-01T00:00:00Z
streams:
variable1: # stream_id
qartod: # StreamConfig
qartod: # Config
location_test:
bbox: [-80, 40, -70, 60]
variable2: # stream_id
qartod: # StreamConfig
qartod: # Config
gross_range_test:
suspect_span: [1, 11]
fail_span: [0, 12]
Expand All @@ -382,7 +382,7 @@ class ContextConfig:
config (odict): dict representation of the parsed ContextConfig source
region (GeometryCollection): A `shapely` object representing the valid geographic region
window (namedtuple): A TimeWindow object representing the valid time period
streams (odict): dict representation of the parsed StreamConfig objects
streams (odict): dict representation of the parsed Config objects
"""

def __init__(self, source: ConfigTypes):
Expand Down
Loading