Steps towards version 1.0

[jsmariegaard]

We are currently re-factoring this library making it ready for version 1.0.

Goals

• Ready to become the validation engine in MOOD (circular stats, dual rose plot)
• Support of more model types (e.g. also machine learning models, WWTP)
• Comparer should have a flexible index - e.g. allow for additional axis
  • Prepare for new geometry types (e.g. vertical profile data or raster data)
  • Prepare for (or support) forecast skill assessment (forecast data would require lead_hour axis as well as time)
• Simplified and consistent api and workflow (e.g. remove Connector, no filtering on plotting functions)
• Simple to make Comparer from already matched data
• Simplified internal structure for easier extension and increased maintainability
• Load/save comparisons to file
• Simpler to filter/aggregate using auxiliary data (co-variates) and custom meta data (attrs)
• Better quantity/unit/iteminfo handling (don't force users to use EUM if not dfs file)
• Should be easy to change names and basic properties after comparison (e.g. if name doesn't look good on plots, it should be possible to change without re-doing comparison from start)

New API

If the modelresult and observation geometry types can easily be inferred (e.g. dfsu or point data):

 import modelskill as ms
 mr1 = ms.model_result(...)
 mr2 = ms.model_result(...)
 o1 = ms.observation(...)
 o2 = ms.observation(...)
 cc = ms.compare([o1, o2], [mr1, mr2])

If the user wants to be more explicit (or geometry types cannot easily be inferred):

 import modelskill as ms
 mr1 = ms.GridModelResult(...)
 mr2 = ms.DfsuModelResult(...)
 o1 = ms.PointObservation(...)
 o2 = ms.TrackObservation(...)
 cc = ms.compare([o1, o2], [mr1, mr2])

If data has already been matched (e.g. resides in a single dfs0/dataframe):

import modelskill as ms
cmp = ms.from_matched(df, obs_item, mod_item, ...)

If the modelresults and observations are specified in a configuration file:

import modelskill as ms
cc = ms.from_config("conf.xlxs")

Tasks:

Week 3

• See documentation section below (prioritize docstrings)
• repr
  • out.append(f" Model: {model}, rmse={self.score()[model]:.3f}") - remove score?
  • Improved repr strings, Useful repr(s) #378
  • SkillGrid repr makes user think it is a xarray! (confusing)
• remove "return getattr" from __getattr__ in both SkillTable and SkillGrid? Delegate less attributes #391
• ask for testing (DACA)

Documentation

• Document recently added features
• DummyModelResult page
• rename API to API Reference (like pandas and xarray), Docs improvements including user guide #363
• Make section: "User Guide" (like pandas and xarray), Docs improvements including user guide #363
  • Selecting/filtering data
• Make section "Examples", Notebook examples in documentation #341
• Decide on tagline and use it consistently!
• Update readme (remove altimetry, add nicer plots) - links to notebooks are no longer valid (after renaming notebooks)
• Remove Taylor Diagram from most of the notebooks (too difficult to understand)
• docstrings on attributes, like data and plot (make them appear in the documentation)
• docstring missing: Comparer.remove_bias
• docstring examples in ...
  • match()
• docstrings: consistent way of showing multiple arg types. Now: [str, int], (str, int) and "str or int" - should be the latter
• add "Trajectory" to description of Track
• Hide subsections like "See Also" in right pane
• Add to issues: Bug: It is not possible to set user-defined default metrics for directional data (will be overridden); it can still be set manually in skill()
• (how to document api of timeseries.plot (it does not appear anywhere in the docs)!, (partly handled by Docs improvements including user guide #363 ))
• (Group notebooks so everything with altimetry is in a subfolder (not to scare non-marine users away))

Bugs/minor

• Prematched_with_auxiliary.ipynb fails! (scatter plot on cmp3) (still fails 2024-1-4) -> now added as issue

Testing

• multi-variable SkillTable with wind speed and wind direction (different metrics)
• Protected names: "Observation" is probably not allowed - test (Result: "Observation" is a valid obs name - it is used in several tests)
• Verify that Comparer/ComparerCollection: rename method allows the user to rename any column (model, obs, aux)
• Verify that No meaningful warning or error when trying to Compare PointModelResult and TrackObservation #220 has been solved
• check problem with combining comparers in WR case notebook

January, 2024, post-release, but soon

• Handling of spatial interpolation (new spatial_interp_method arg?)
• Timeseries plot time axis appearance not nice if non-equidistant calendar, Consistent and concise datetime axis on timeseries plots (matplotlib) #347
• Color handling: each model should have a unique color which should be assigned when creating comparer. (re-surface color property?)
• attrs on model could be really useful for calibration (especially if calibrating several params over many runs); allow user to provide attrs on MR and filter/groupby on model attrs
• DfsuModelResult and GridModelResult misses properties (names, quantities, time)
• Check how many changes are needed to handle 2d or 3d index (e.g. time,z or time,lead_hour or time,x,y) (test by xr.expand_dims)
• allow from_matched to take xr.Dataset as input
• cc.plot.temporal_coverage()
• How do we make it simple to change quantity (e.g. unit) (Make quantity mutable to it is easy to change long_name etc; postpone)
• SkillArray.plot should have a plot.map and/or plot.explore function
• SkillGrid could have quantity property for consistency
• sel() method should use part of e.g. model name as long as it is unique
• sel() method should allow use of "*" (like in mikeio)
• drop() method to drop obs/mod/variable (reverse sel)
• (API: should comparers have both query() and where()? )
• Scatter ax argument in case of multiple models (=multiple plots)
• cc.plot: box plot is missing! (currently only Comparer supports box plot)
• cc.plot: qq and residual_hist missing! (currently only in Comparer - hence inconsistent)
• (Add name argument in from_matched() and/or match(), already so???)
• Methods in SkillTable could be moved to SkillMixIn (e.g. sort_values) and then use in both SkillTable and SkillArray
• point/grid should take .nc files similar to GridModelResult
• Time interpolation of directional data - avoid interpolating over 360-0.
• Check spatial match when comparing point to point (new spatial_tolerance arg in match?)
• add z to _to_long_dataframe() if not None or np.nan

Refactoring

• ItemSelection when reading files: create _utils module in _timeseries: Refactor the way we select/validate items. It is now quite similar in Observation and ModelResults - track is a bit different (also has x and y item), but the rest is the same. This should be refactored out. Postponed from Allow aux items in ModelResults #304

• Comparer/Timeseries: A way to bypass dataset validation if a result of filtering
• groupby code in skill methods

Considerations/thoughts

• How to handle weighted metrics (sample_weights)? (use "weights" in inspect.signature(metric).parameters to find out if sample_weights should be applied in a specific)
• Should we allow np.nan in Comparer (maybe only drop if necessary)
• How should empty comparers be handled (e.g. upon filtering)?
• What to do with empty comparers? Allowed? Not-allowed? Throw error? (e.g. if no overlapping data in matching)
• Create example file structure (how we suggest you organize your files)

Naming

• Should we rename swap_level in skill table?
• Should we use name "Dfsu", "FlexibleMesh" or "Unstructured"?
• Consistent naming: mod_ or model_ or mr or modelresults
• Should we rename items to avoid spaces, commas etc - makes it difficult to use query() etc
• Use variable or quantity (or both)

Long-list, postponed, 2024

• isel() method? Comparer/ComparerCollection isel #374
• SkillGrid could have to_geodataframe() like SkillTable
• Comparer/ComparerCollection plotting with multiple models
  • cmp.plot.scatter() behaviour - different markers, BW vs colorscale, multiple axes?
• Forecast skill: Interpolation only in forecast direction, 2024
• Idea: Polygon Skill - skill aggregated over a basin (assign observations to group ...)
• How to include non-metric analytics (max, std etc) in skill table? (could add as an example, if SkillTable is added to root namespace)
• (AggregatedSkill: Change data container to xarray dataset, postponed)
• (Remove Connector when config, plots etc have been moved. Update tests and examples accordingly. ) wait until version 1.1
• (Should the skill() method return a SkillArray instead of a SkillTable if metrics is a str?, postponed/dropped)
• re-establish mean_skill_points() (currently disabled as no longer correct)
• Should we allow users to merge two SkillTable objects? (e.g. if one contains weighted and another not)
• sk.style of new/own metrics

Done

• Check for use of "id" where "idx" is really meant
• Make a new notebook with WWTP example, Addition of Wastewater notebook example and dataset #336
• API split into many smaller files, grouped
• Perf: use categorical vars instead of strings to identify obs/model in long dataframe view in ComparerCollection. -> faster groupby, Skill groupby attrs #351
• Introduce new spatial_interp argument?
• Skill groupby: sort=False, Don't sort on groupby (e.g. in skill() method) #376
• Bug: Skill groupby - set track observation x, y to NaN, Don't sort on groupby (e.g. in skill() method) #376
• Re-introduce cmp.to_dataframe() method after renaming existing internal func (long format), Don't sort on groupby (e.g. in skill() method) #376
• Release final beta, Monday, January 8
• API: Make SkillTable accessible from root name space and init should take both df and xr.ds, SkillTable in root namespace #372
• API: Remove more properties: Remove/deprecate more properties #373
  • Collection has a few properties that are undocumented and maybe should be removed or made private? name, unit_text, n_comparers, n_observations (maybe keep for consistency)?
  • geometry property (shapely) on obs objects? (or rename to to_shapely() )
• API: plotting.scatter take a skill_df argument which is inconsistent with cc.plot.scatter
  • scatter plot should NOT take skill_df but a dict (see e.g. format_skill_df) skill_table also for plotting.scatter #369
• API: Scatter plot argument list are slightly different in Comparer vs ComparerCollection! Difference: "norm" and "ax". Scatter consistent args #370
• ms.from_config still returns connector! Should return ComparisonCollection now!, from_config() return ComparerCollection #366
• Bug: rename Observation name does not work in comparer!, Rename enhanced #368
• Protected names: "time", "x", "y", "z", "Observation" can not be used for naming observations and modelresults; add check (also to rename method), Rename enhanced #368
• API: rename variable to quantity (e.g. in sel() method), Rename variable to quantity #365
• Bug: fix random colors in observations (decide on fixed color for observation, unless defined by user), Fix random obs colors #367
• skill() groupby attrs, Skill groupby attrs #351
• API: Make Comparer/Collection to_dataframe method internal: _to_long_dataframe(), Skill groupby attrs #351
• Bug: multi-model scatter plots show same skill table!!
• API: remove more properties from classes, e.g. mod, Slim public API; remove properties #354
• skill() groupby Month-in-Year etc, Skill groupby dt #352
• Should we remove "weight" as an attribute on observation/comparer (only used in a single method: mean_skill() )... ? Use observation weights in score #342
• API: Re-establish weighted observations? (currently broken also color) or should it just be on mean skill - see consideration below, Use observation weights in score #342
• Bug: box plot shows x,y, z as if they were variables
• Any risks/problems with "observation" which is both the name of a root-namespace function and a module!?
• API: consistency: end (comparer) or end_time (timeseries): TODO: remove except on cc (rename to end_time), Remove start end properties #353
• API: consistency: field_names (SkillGrid) or metrics (SkillTable)
• Rename SpatialSkill to GriddedSkill or SkillGrid, Rename SpatialSkill to GriddedSkill #315
• Allow compare to return Comparer if observation is str (more consistent with from_matched)
• Make sure all tests and examples etc use "import modelskill as ms"
• Remove all usage of Connector
• Make a new notebook with urban drainage example, Add a event-based example #274
• Make a new notebook with water resource example (river network), First water resource test case #249
• AggregatedSkill: split in a dataset and a dataarray part, so that you do not need to specify "metric" in the plot functions, SkillTable is a collection of SkillArrays #309
• ms.observation() function similar to ms.model_result(), Observation factory function #328
• CHGA: score() function should return a dictionary if multiple models (not clear enough which score belongs to which model) - already so for collection, Ensure Comparer / ComparerCollection consistency #330
• Comparer.plot: multiple models should lead to multiple plots (for now - later we can add options), Scatter plot return multiple plots for multiple models #337
• hist plot still has model argument -> return multiple plots instead, Scatter plot return multiple plots for multiple models #337
• mean_skill_points() no longer correct! -> out-comment for now
• SkillGrid sel() method to avoid "model" argument in plot() method, Skillgrid.sel #340
• remove_bias() should return a new Comparer object instead of modifying self
• Circular statistics (e.g. wave direction), Circular metrics (again) #284
• ItemInfo to long_name and unit (i.e. if input is dfs file make sensible conversion to long_name + unit AND change plots/tables to generate labels based on long_name and unit instead of itemInfo) Replace ItemInfo with fmskill.Quantity #191
• GridSkill: split in a dataset and a dataarray part, so that you do not need to specify "metric" in the plot functions
• Clean-up use of types, protocols and base classes - should be consistent across the library and easy to understand and work with in the future, @ecomodeller
• Clean-up: e.g. rename all data fields to .data.
• Remove multi-item modelresults.
• Change class structure of ModelResults such that they Resemble Observations: PointModelResult, TrackModelResult etc.
• Create new internal data structure .data for SingleObsComparer and other Comparers based on xarray.
• Implement sel() method.
• Change internal data structure of ModelResults and Observations to use xarray dataset (make it simpler to carry over meta data to Comparer)
• Allow both ModelResults and Observations to carry aux data (aux_item argument)
  • Example: upon skill assessment of wave period (Tp) I want to make it conditional on wave height (Hm0). The ModelResult has that information and modelskill could extract Hm0 from the dfsu file together with Tp. Or maybe the measurement file has that information and we would like to get it from there. (The user would perhaps like to assign a custom name to the aux variable e.g. "obs_Hm0"), Allow aux items in ModelResults #304
• Allow custom metadata from Observations (and maybe ModelResult) in an attrs container to be carried forward to the Comparer.
  • Example: in data assimilation cases, I would like to be able to distinguish between "DA" and "validation" stations. In order cases, it could be different "tanks" in a WWTP or offshore/nearshore stations, Observation attrs container #291
    generate labels based on long_name and unit instead of itemInfo) Replace ItemInfo with fmskill.Quantity #191
• Rich compare() method (removing the need for the connector) Rich compare method #180
• Move from_config() out of Connector and into new module configuration.py or configurator.py (how to handle to_config? (method in root name space and/or method on ComparerCollection?))
• Move plot_temporal_coverage() and plot_observation_positions() to root name space move plot methods from Connector to root namespace #187
• Deprecate Connector by issuing deprecation warnings.
• Load/save functionality in Comparer and ComparerCollection Save/load comparer(s) #178
• New function ms.from_matched(data, obs_item, mod_item) Matched data in a single dataframe #192
• New function validate_comparer_dataset (to check the consistency of the xarray dataset that constitutes Comparer)
• Rename package to modelskill
• Update all examples, notebooks and tests to use new api (import modelskill as ms, ...)
• Use new logo new logo #184
• Make pre-release of modelskill to pypi to reserve the name
• Check how many changes are needed to handle data without time axis? Time is not essential for skill assessment #293
• Handling of circular variables (also plots etc - grid at 0, 90, 180, 270 and 360)
• Dual-rose plot for comparison of two circular data sets, @daniel-caichac-DHI
• Loosen requirement in Comparer that dataset must have a DateTimeIndex, Time is not essential for skill assessment #293

[Hendrik1987]

@jsmariegaard : This seems very exciting!
Are you planning on some pre-release of 1.0? Would be keen to try it out on a project.
Cheers!

[jsmariegaard]

@Hendrik1987, the first alpha is already on PyPI: https://pypi.org/project/modelskill/ but this is mostly to reserve the name. Proper pre-releases will be made soon after the summer in August/September.