Implement `entropy` dispatches for distributions

[jessegrabowski]

Description

#7530 added a helper to compute the closed-form mean of a distribution, and #8084 adds variance.

One last useful helper that pops up in applications is the entropy. It's used in ML as a regularization term (e.g. in reinforment learning models that learn a policy function like DDPG), and can be used to pick priors (e.g. in pz.maxent.

It could be nice to have these around. I'm marking as request discussion because I realize by adding all these different moments and statistics we're going down a bit of a rabbit hole, and maybe someone wants to step in and say stop now.

[aloctavodia]

All distributions here https://github.com/pymc-devs/distributions have entropy, variance, and other moments and stuff. Is that what you need? Or you are thinking of a different approach?

[jessegrabowski]

That might be a better home for all of this. Did you look at what was being done in the two PRs I linked? The difference is that we're adding a pm.mean and pm.variance that can take a random variable as input, extract the symbolic parameters, then do the formulas.

It would be trivial to have such dispatches on top of what you have in the linked code

[jessegrabowski]

The other option would be to make distributions an optional dependency, keep the dispatch helpers in pymc, calling the relevant function from distributions

[ricardoV94]

The idea with distributions package is to offload the pytensor code so it can be used from other libraries without requiring PyMC as a dependency. PyMC should switch to using it.

The derived stuff will remain PyMC specific, but at the end of the day those always build on top of something basic which will live in the distributions package.

Basically the logp of pm.Normal will be a dispatch for something in the distributions package, instead of an expression that exists in the pymc codebase. logp of pt.exp(pm.Normal), will be a mix of both

[jessegrabowski]

So I should change #8084 to use distributions

[ricardoV94]

I don't think distributions is released yet? So if you need this functionality soon go ahead. Otherwise the long term path is through there.

[jessegrabowski]

Not urgent but distributions looks pretty good. Is there any hold-up to a v0 release?

[ricardoV94]

@aloctavodia ?

[aloctavodia]

This is the list of distributions currently implemented pymc-devs/pytensor-distributions#3 at this point the ones missing reflect the ones needed to fully replace preliz's distributions. So compared with distributions in PyMC that list is missing some multivariate ones, and stuff like Flat, timeseries...
I can work next week on adding more distributions @fonnesbeck and @juanitorduz have been helping too.

[Jayant-kernel]

Hey! I'd love to help out with this if it's still available.

I've looked at the mean() and variance() implementations in #7530 and #8084, and the pattern makes sense. I'm thinking I'd follow the same single-dispatch approach - start with a few easy distributions (Normal, Uniform, Exponential), then expand from there.

Quick question though - you mentioned worrying about going down a rabbit hole with these helpers. Should entropy be the stopping point, or are there other statistics you'd find useful? Just want to make sure this aligns with where the project's headed.

Happy to take this on if you're good with it!

[jessegrabowski]

Hi Jayant,

I think this is on hold pending the v0 release of pytensor-distributions, then we can do a PR that can implement entropy as thing wrappers around functions implemented in that library.