This repository has been archived by the owner on Feb 16, 2022. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 8
Add Usage Recommendations section #18
Merged
Merged
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
# Usage Recommendations Fragment | ||
|
||
This fragment describes conditions under which the model publishers believe the model will perform | ||
as expected. | ||
|
||
While the [Model Training] section gives users a clear picture of the training conditions, the model | ||
publisher may also have evidence or intuition that the model will also perform well when used in a | ||
different context. For instance, a model might have been trained on data from a particular | ||
geographic region and time period, but based on some combination of intuition, previous experience, | ||
and informal experimentation, the model publisher may be confident that it will perform similarly in | ||
a new geographic region. This section is meant to capture that information in a systematic way, | ||
while also providing publishers the opportunity to describe their recommendations in a more | ||
free-form style. | ||
|
||
## Directory Contents | ||
|
||
* [`usage-fragment.md`](./usage-fragment.md) - The detailed Usage Recommendations Fragment | ||
specification. | ||
|
||
[Model Training]: ../training |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,80 @@ | ||
# Usage Recommendations Specification | ||
|
||
The Usage Recommendations section defines conditions under which the model publishers believe the | ||
model will perform as expected. It also provides a mechanism for publishers to explicitly call out | ||
conditions under which they believe the model is likely to not performa as expected. It is important | ||
to note that these are *recommendations* and that they **in no way provide any guarantee of model | ||
performance under the given conditions**. | ||
|
||
If no Usage Recommendation section is provided, then users should assume that the model will only | ||
perform as expected under conditions that match those described in the [Model Training] section. | ||
This also applies to any fields that are not present in the Usage Recommendations section. For | ||
instance, if no `temporal` field is provided, users should assume the model will only perform as | ||
expected for data in the time range given in the [Model Training] section. | ||
|
||
|
||
## Usage Recommendations Fields | ||
|
||
The `recommendations` and `cautions` fields in this section are arrays of objects that describe | ||
conditions under which the model either should perform as expected or should *not* behave as | ||
expected, respectively. | ||
|
||
If the Usage Recommendations section is present, then at least one of these fields is required and | ||
must be a non-empty array. | ||
|
||
| Field Name | Type | Description | | ||
| ----------------- | --------------------- | ---------------------------------------------------------------------------------------------- | | ||
| `recommendations` | \[[Usage Conditions]] | An array of objects describing conditions under which the model *should* perform as expected. | | ||
| `cautions` | \[[Usage Conditions]] | An array of objects describing conditions under which the model *may not* perform as expected. | | ||
|
||
## Usage Conditions | ||
|
||
This object is used to described conditions under which model usage is either recommended (if found | ||
in the `recommendations` list) or discouraged (if found in the `cautions` list). | ||
|
||
| Field Name | Type | Description | | ||
| ----------- | ------------------------ | ------------------------------------------------------------------------------------------ | | ||
| spatial | [Spatial Extent Object] | The spatial extents to which the recommendation or caution applies. | | ||
| temporal | [Temporal Extent Object] | The temporal extents to which the recommendation or caution applies. | | ||
| description | string | A human-readable description of conditions to which the recommendation or caution applies. | | ||
|
||
### Spatial Extent Object | ||
|
||
| Element | Type | Description | | ||
| ------- | ------------ | -------------------------------------------------------------------------------------------- | | ||
| bbox | \[\[number]] | **REQUIRED.** Potential *spatial extents* within which the model should perform as expected. | | ||
|
||
The `bbox` element is an *array* whose elements are themselves arrays. Each inner array is a bounding | ||
box that describes a spatial extent within which the model should perform as expected. | ||
|
||
The format of the inner bounding box arrays follows the specification outlined in the **bbox** | ||
section of the [STAC Spatial Extent Object] documentation. However, the `bbox` element defined in | ||
this spec differs from the `bbox` element in the STAC Spatial Extent Object in that there is no | ||
first element that always represents the overall spatial extent. Such an overall extent might give | ||
users the incorrect impression that the model should perform as expected anywhere within that | ||
spatial extent, when in fact there may be only specific areas in which this is true. | ||
|
||
### Temporal Extent Object | ||
|
||
| Element | Type | Description | | ||
| -------- | ------------------ | --------------------------------------------------------------------------------------------- | | ||
| interval | \[\[string\|null]] | **REQUIRED.** Potential *temporal extents* within which the model should perform as expected. | | ||
|
||
The `interval` element is an *array* whose elements are themselves arrays. Each inner array is an | ||
interval array containing 2 elements, each of which may be either `null` or a timestamp. Timestamps | ||
must follow the specifications outlined in the [STAC Temporal Extent Object] documentation. | ||
|
||
The format of the inner interval arrays follows the specification outlined in the [STAC Temporal | ||
Extent Object] documentation. However, the `interval` element defined in this spec differs from | ||
the `interval` element in the STAC Temporal Extent Object in that there is no first element that | ||
always represents the overall temporal extent. Such an overall extent might give users the | ||
incorrect impression that the model should perform as expected anywhere within that temporal range, | ||
when in fact there may be only specific ranges in which this is true. | ||
|
||
|
||
[Model Training]: ../training | ||
[Spatial Extent Object]: #spatial-extent-object | ||
[Temporal Extent Object]: #temporal-extent-object | ||
[Usage Conditions]: #usage-conditions | ||
[STAC Spatial Extent Object]: https://github.com/radiantearth/stac-spec/blob/v1.0.0-rc.4/collection-spec/collection-spec.md#spatial-extent-object | ||
[STAC Temporal Extent Object]: https://github.com/radiantearth/stac-spec/blob/v1.0.0-rc.4/collection-spec/collection-spec.md#temporal-extent-object |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A short sentence/paragraph could be added, noting that the section can also provide possible conditions, where the model will not perform.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated in a829170 to have separate
recommendations
andcautions
sections.