Bump torchmetrics from 0.9.3 to 0.11.1

[dependabot]

Bumps torchmetrics from 0.9.3 to 0.11.1.

Release notes

Sourced from torchmetrics's releases.

Minor patch release

[0.11.1] - 2023-01-30

Fixed

• Fixed type checking on the maximize parameter at the initialization of MetricTracker (#1428)
• Fixed mixed precision auto-cast for SSIM metric (#1454)
• Fixed checking for nltk.punkt in RougeScore if a machine is not online (#1456)
• Fixed wrongly reset method in MultioutputWrapper (#1460)
• Fixed dtype checking in PrecisionRecallCurve for target tensor (#1457)

Contributors

@​borda, @​SkafteNicki, @​stancld

If we forgot someone due to not matching commit email with GitHub account, let us know :]

Full Changelog: Lightning-AI/torchmetrics@v0.11.0...v0.11.1

Adding Multimodal and nominal domain

We are happy to announce that Torchmetrics v0.11 is now publicly available. In Torchmetrics v0.11 we have primarily focused on the cleanup of the large classification refactor from v0.10 and adding new metrics. With v0.11 are crossing 90+ metrics in Torchmetrics nearing the milestone of having 100+ metrics.

New domains

In Torchmetrics we are not only looking to expand with new metrics in already established metric domains such as classification or regression, but also new domains. We are therefore happy to report that v0.11 includes two new domains: Multimodal and nominal.

Multimodal

If there is one topic within machine learning that is hot right now then it is generative models and in particular image-to-text generative models. Just recently stable diffusion v2 was released, able to create even more photorealistic images from a single text prompt than ever

In Torchmetrics v0.11 we are adding a new domain called multimodal to support the evaluation of such models. For now, we are starting out with a single metric, the CLIPScore from this paper that can be used to evaluate such image-to-text models. CLIPScore currently achieves the highest correlation with human judgment, and thus a high CLIPScore for an image-text pair means that it is highly plausible that an image caption and an image are related to each other.

Nominal

If you have ever taken any course in statistics or introduction to machine learning you should hopefully have heard about data can be of different types of attributes: nominal, ordinal, interval, and ratio. This essentially refers to how data can be compared. For example, nominal data cannot be ordered and cannot be measured. An example, would it be data that describes the color of your car: blue, red, or green? It does not make sense to compare the different values. Ordinal data can be compared but does have not a relative meaning. An example, would it be the safety rating of a car: 1,2,3? We can say that 3 is better than 1 but the actual numerical value does not mean anything.

In v0.11 of TorchMetrics, we are adding support for classic metrics on nominal data. In fact, 4 new metrics have already been added to this domain:

• CramersV
• PearsonsContingencyCoefficient
• TschuprowsT
• TheilsU

All metrics are measures of association between two nominal variables, giving a value between 0 and 1, with 1 meaning that there is a perfect association between the variables.

Small improvements

In addition to metrics within the two new domains v0.11 of Torchmetrics contains other smaller changes and fixes:

• TotalVariation metric has been added to the image package, which measures the complexity of an image with respect to its spatial variation.

• MulticlassExactMatch metric has been added to the classification package, which for example can be used to measure sentence level accuracy where all tokens need to match for a sentence to be counted as correct

... (truncated)

Changelog

Sourced from torchmetrics's changelog.

[0.11.1] - 2023-01-30

Fixed

• Fixed type checking on the maximize parameter at the initialization of MetricTracker (#1428)
• Fixed mixed precision autocast for SSIM metric (#1454)
• Fixed checking for nltk.punkt in RougeScore if a machine is not online (#1456)
• Fixed wrongly reset method in MultioutputWrapper (#1460)
• Fixed dtype checking in PrecisionRecallCurve for target tensor (#1457)

[0.11.0] - 2022-11-30

Added

• Added MulticlassExactMatch to classification metrics (#1343)
• Added TotalVariation to image package (#978)
• Added CLIPScore to new multimodal package (#1314)
• Added regression metrics:
  • KendallRankCorrCoef (#1271)
  • LogCoshError (#1316)
• Added new nominal metrics:
  • CramersV (#1298)
  • PearsonsContingencyCoefficient (#1334)
  • TschuprowsT (#1334)
  • TheilsU (#1337)
• Added option to pass distributed_available_fn to metrics to allow checks for custom communication backend for making dist_sync_fn actually useful (#1301)
• Added normalize argument to Inception, FID, KID metrics (#1246)

Changed

• Changed minimum Pytorch version to be 1.8 (#1263)
• Changed interface for all functional and modular classification metrics after refactor (#1252)

Removed

• Removed deprecated BinnedAveragePrecision, BinnedPrecisionRecallCurve, RecallAtFixedPrecision (#1251)
• Removed deprecated LabelRankingAveragePrecision, LabelRankingLoss and CoverageError (#1251)
• Removed deprecated KLDivergence and AUC (#1251)

Fixed

• Fixed precision bug in pairwise_euclidean_distance (#1352)

[0.10.3] - 2022-11-16

Fixed

• Fixed bug in Metrictracker.best_metric when return_step=False (#1306)
• Fixed bug to prevent users from going into an infinite loop if trying to iterate of a single metric (#1320)

... (truncated)

Commits

• 8b3253e release 0.11.1
• c60b912 Fix checking for nltk.punkt if a machine not online (#1456)
• d4607ee Return info / formatting on Classification docs (#1433)
• f884c0e Fix documentation source links (#1439)
• be7e4e7 Add output / format input info to retrieval docs (#1429)
• 2a6bdf5 ci: extend GPU timeout
• 1797bf4 Fix autocast in SSIM metric (#1454)
• fca8d74 Change Python3 in pre-commit-config.yaml (#1463)
• 09131f2 Fix broken reset method in MultioutputWrapper (#1460)
• b45213e Add missing dtype check in PrecisionRecallCurve (#1457)
• Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge will merge this PR after your CI passes on it
• @dependabot squash and merge will squash and merge this PR after your CI passes on it
• @dependabot cancel merge will cancel a previously requested merge and block automerging
• @dependabot reopen will reopen this PR if it is closed
• @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

[dependabot]

Superseded by #52.