Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Broadcasting probabilistic Unaggregated measures (such as proper scoring rules) over a single prediction #635

Open
ablaom opened this issue Sep 10, 2021 · 0 comments

Comments

@ablaom
Copy link
Member

ablaom commented Sep 10, 2021

The idea of implementing "distribution fitters" as supervised models is to enable their evaluation using a proper scoring rule (which in turn makes hyper-parameter optimization possible).

There are precisely zero models implementing this API, but as currently specified, predict(mach, nothing) returns the fitted distribution. Now to evaluate using some examples using, say, log_score, we need a broadcast version of log_score(yhat, y) where yhat is a single distribution and y a number of observations (samples). Currently measures expect yhat and y to be arrays of the same dimension.

@ablaom ablaom added invalid This doesn't seem right version 1.0 and removed invalid This doesn't seem right labels Sep 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant