Implement negative binomial

[raphtael]

Hi, I work in ecology where we often analyse count data. While in theory the default distribution for count data is the Poisson, in practice the data is more often than not overdispersed relative to the Poisson distribution. In this case, we default to using GLMs with a negative binomial likelihood. I am not sure how difficult the negative binomial likelihood would be to implement in projpred, but I can see it to be a useful addition for many users. To give an example, I am currently working on a project where I have around 2500 count responses and 350 potential predictors. I expect maybe 5-10% of the predictors to be associated with my response. In this scenario, fitting a reference model with horseshoe priors and doing variable selection with projpred seems to be the right thing to do. However, since my response is overdispersed, it seems to be more appropriate to use a negative binomial likelihood instead of a Poisson. Thanks for your consideration.

[AlejandroCatalina]

Sorry I took some time to get back to this because of my Christmas holidays!

We are pursuing this direction as we speak and already have a prototype to deal with models outside of the exponential family. I hope I can bring more news soon for you to test it.

[StaffanBetner]

In the meanwhile I can suggest that you fit a Poisson model with a random intercept per observation, which also models the overdispersion. The difference to a negative binomial model is that the additional variation is assumed to be normally distributed instead of Gamma distributed (since the negative binomial distribution is a Poisson-Gamma mixture).

[fweber144]

See also #361.

[fweber144]

Forgot to mention here that the latent projection can be used for negative binomial models, see the latent vignette. Otherwise, #361 tentatively describes how the projection problem could be solved exactly in case of the negative binomial family, so closing this issue here now.