Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

load_collection and load_stac: Clarify the dimension names and labels #491

Open
wants to merge 3 commits into
base: draft
Choose a base branch
from
Open

load_collection and load_stac: Clarify the dimension names and labels #491

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e31192d
`load_collection` and `load_stac`: Clarify the order of dimension lab…
m-mohr Jan 3, 2024
c03ee1e
`load_collection` and `load_stac`: Clarify the dimension naming. #489
m-mohr Jan 3, 2024
4460104
Remove "expected to be"
m-mohr Jan 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- `eq` and `neq`: Explicitly set the minimum value for the `delta` parameter.
- `filter_bbox`, `load_collection`, `load_stac`: Clarified that the bounding box is reprojected to the CRS of the spatial data cube dimensions if required.
- `filter_spatial`: Clarified that masking is applied using the given geometries. [#469](https://github.com/Open-EO/openeo-processes/issues/469)
- `load_collection` and `load_stac`: Clarify the dimension naming and the order of dimension labels for nominal labels. [#488](https://github.com/Open-EO/openeo-processes/issues/488), [#489](https://github.com/Open-EO/openeo-processes/issues/489)
- `mod`: Clarified behavior for y = 0
- `sqrt`: Clarified that NaN is returned for negative numbers.

Expand Down
4 changes: 2 additions & 2 deletions load_collection.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"id": "load_collection",
"summary": "Load a collection",
"description": "Loads a collection from the current back-end by its id and returns it as a processable data cube. The data that is added to the data cube can be restricted with the parameters `spatial_extent`, `temporal_extent`, `bands` and `properties`. If no data is available for the given extents, a `NoDataAvailable` exception is thrown.\n\n**Remarks:**\n\n* The bands (and all dimensions that specify nominal dimension labels) are expected to be ordered as specified in the metadata if the `bands` parameter is set to `null`.\n* If no additional parameter is specified this would imply that the whole data set is expected to be loaded. Due to the large size of many data sets, this is not recommended and may be optimized by back-ends to only load the data that is actually required after evaluating subsequent processes such as filters. This means that the values in the data cube should be processed only after the data has been limited to the required extent and as a consequence also to a manageable size.",
"description": "Loads a collection from the current back-end by its id and returns it as a processable data cube. The data that is added to the data cube can be restricted with the parameters `spatial_extent`, `temporal_extent`, `bands` and `properties`. If no data is available for the given extents, a `NoDataAvailable` exception is thrown.\n\n**Remarks:**\n\n* All dimensions that specify nominal dimension labels (e.g. bands) are expected to be ordered as specified in the data cube metadata (`cube:dimensions`) unless otherwise specified in a corresponding parameter (e.g. `bands`).\n* If no additional parameter is specified this would imply that the whole data set is expected to be loaded. Due to the large size of many data sets, this is not recommended and may be optimized by back-ends to only load the data that is actually required after evaluating subsequent processes such as filters. This means that the values in the data cube should be processed only after the data has been limited to the required extent and as a consequence also to a manageable size.",
"categories": [
"cubes",
"import"
Expand Down Expand Up @@ -159,7 +159,7 @@
},
{
"name": "bands",
"description": "Only adds the specified bands into the data cube so that bands that don't match the list of band names are not available. Applies to all dimensions of type `bands`.\n\nEither the unique band name (metadata field `name` in bands) or one of the common band names (metadata field `common_name` in bands) can be specified. If the unique band name and the common name conflict, the unique band name has a higher priority.\n\nThe order of the specified array defines the order of the bands in the data cube. If multiple bands match a common name, all matched bands are included in the original order.\n\nIt is recommended to use this parameter instead of using ``filter_bands()`` directly after loading unbounded data.",
"description": "Only adds the specified bands into the data cube so that bands that don't match the list of band names are not available. Applies to all dimensions of type `bands`.\n\nEither the unique band name (metadata field `name` in bands) or one of the common band names (metadata field `common_name` in bands) can be specified. If the unique band name and the common name conflict, the unique band name has a higher priority.\n\nThe order of the specified array defines the order of the bands in the data cube. If multiple bands match a common name, all matched bands are included in the original order. If the parameter is set to `null`, the order of the bands is as defined for the bands dimension in the data cube metadata (metadata field `cube:dimensions`).\n\nIt is recommended to use this parameter instead of using ``filter_bands()`` directly after loading unbounded data.",
"schema": [
{
"type": "array",
Expand Down
4 changes: 2 additions & 2 deletions proposals/load_stac.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"id": "load_stac",
"summary": "Loads data from STAC",
"description": "Loads data from a static STAC catalog or a STAC API Collection and returns the data as a processable data cube. A batch job result can be loaded by providing a reference to it.\n\nIf supported by the underlying metadata and file format, the data that is added to the data cube can be restricted with the parameters `spatial_extent`, `temporal_extent` and `bands`. If no data is available for the given extents, a `NoDataAvailable` exception is thrown.\n\n**Remarks:**\n\n* The bands (and all dimensions that specify nominal dimension labels) are expected to be ordered as specified in the metadata if the `bands` parameter is set to `null`.\n* If no additional parameter is specified this would imply that the whole data set is expected to be loaded. Due to the large size of many data sets, this is not recommended and may be optimized by back-ends to only load the data that is actually required after evaluating subsequent processes such as filters. This means that the values should be processed only after the data has been limited to the required extent and as a consequence also to a manageable size.",
"description": "Loads data from a static STAC catalog or a STAC API Collection and returns the data as a processable data cube. A batch job result can be loaded by providing a reference to it.\n\nIf supported by the underlying metadata and file format, the data that is added to the data cube can be restricted with the parameters `spatial_extent`, `temporal_extent` and `bands`. If no data is available for the given extents, a `NoDataAvailable` exception is thrown.\n\n**Remarks:**\n\n* The dimensions (e.g. names) follow the data cube metadata (`cube:dimensions`), if present. Otherwise, it tries to preserve any dimension names available in the files. Otherwise, it falls back to the recommended dimension names as specified in the openEO API (`x`, `y`, `z`, `t`, `bands`, `geometry`).\n* All dimensions that specify nominal dimension labels (e.g. bands) are expected to be ordered as specified in the data cube metadata (`cube:dimensions`) unless otherwise specified in a corresponding parameter (e.g. `bands`). If no data cube metadata is present, the order of the dimension labels will reflect the structure in the files.\n* If no additional parameter is specified this would imply that the whole data set is expected to be loaded. Due to the large size of many data sets, this is not recommended and may be optimized by back-ends to only load the data that is actually required after evaluating subsequent processes such as filters. This means that the values should be processed only after the data has been limited to the required extent and as a consequence also to a manageable size.",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If no data cube metadata is present, the order of the dimension labels will reflect the structure in the files.

Isn't it safer to say that the order will be undefined?
Otherwise users might be tricked into depending on implementation details that can suddenly change. Also, "structure in the files" might be not as straightforward practically as it sounds (e.g. multiple files where band order is different).

If a predictable order is important, we might also prescribe to use alphabetical order if no other reliable band order source is available.

Copy link
Member Author

@m-mohr m-mohr Jan 4, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For us, maybe. For the user, undefined is the worst case scenario, I think.

load_stac is anyway implementation heavily implementation detail dependant, the order of the bands is just a tiny detail of it. If you change band order in the source files, you may change a lot more (think of recent S2 changes with the offsets etc). So I don't see an issue here (at least not only for bands).

Alphabetical order as in B1, B10, A2 or B1, B2, B10? ;-)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the user, undefined is the worst case scenario, I think.

I'm not sure. I'd rather prefer an simple honest statement that the order is undefined than getting the impression that the order is static (unless you understand the fine print with a lot of technical jargon).

But anyway, I guess the main message should be: dear user, if you care about the band order, specify it explicitly

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and if alphabetical is an option I would keep it simple to pure alphabetical: B1, B10, B2. The main goal is to be predictable, not trying to guess what makes most sense for humans.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I fully agree with Stefaan. And having an explicit statement telling that for some edge cases the order can't be defined automatically and the best practice consist in defining explicitly the bands to load and their order.

"categories": [
"cubes",
"import"
Expand Down Expand Up @@ -165,7 +165,7 @@
},
{
"name": "bands",
"description": "Only adds the specified bands into the data cube so that bands that don't match the list of band names are not available. Applies to all dimensions of type `bands`.\n\nEither the unique band name (metadata field `name` in bands) or one of the common band names (metadata field `common_name` in bands) can be specified. If the unique band name and the common name conflict, the unique band name has a higher priority.\n\nThe order of the specified array defines the order of the bands in the data cube. If multiple bands match a common name, all matched bands are included in the original order.\n\nIt is recommended to use this parameter instead of using ``filter_bands()`` directly after loading unbounded data.",
"description": "Only adds the specified bands into the data cube so that bands that don't match the list of band names are not available. Applies to all dimensions of type `bands`.\n\nEither the unique band name (metadata field `name` in bands) or one of the common band names (metadata field `common_name` in bands) can be specified. If the unique band name and the common name conflict, the unique band name has a higher priority.\n\nThe order of the specified array defines the order of the bands in the data cube. If multiple bands match a common name, all matched bands are included in the original order. If the parameter is set to `null`, the order of the bands is as defined for the bands dimension in the data cube metadata (metadata field `cube:dimensions`).\n\nIt is recommended to use this parameter instead of using ``filter_bands()`` directly after loading unbounded data.",
"schema": [
{
"type": "array",
Expand Down