Skip to content

Dependence aware tests for ppc*ecdf#428

Open
florence-bockting wants to merge 152 commits intostan-dev:masterfrom
florence-bockting:dependence-aware-LOO-PIT
Open

Dependence aware tests for ppc*ecdf#428
florence-bockting wants to merge 152 commits intostan-dev:masterfrom
florence-bockting:dependence-aware-LOO-PIT

Conversation

@florence-bockting
Copy link
Copy Markdown

@florence-bockting florence-bockting commented Mar 4, 2026

Description

The current approach in ppc_loo_pit_ecdf and ppc_pit_ecdf assumes independence of LOO-PIT values which is not valid (Marhunenda et al., 2005). The corresponding graphical test yields an envelope that is too wide, reducing the test's ability to reveal model miscalibration.
Tesso & Vehtari (2026, see preprint) propose three testing procedures that can handle any dependent uniform values and provide an updated graphical representation that uses color coding to indicate influential regions or most influential points of the ECDF. This PR implements the new development, by adding the updated approach (method = "correlated") additionally to the previous approach (method = "independent").

TODOs

  • updated ppc_loo_pit_ecdf() function in ppc-loo.R
  • updated ppc_pit_ecdf() and ppc_pit_ecdf_grouped() function in ppc-distributions.R
  • add unittests and visual regression tests
  • update documentation
  • deprecation suggestion of old method
    • P1: old method (default) and new method available via method argument
    • P2: new method (default) and old method available via method argument
    • P3: old method is removed

@florence-bockting florence-bockting changed the title Dependence aware tests for ppc_loo_pit_ecdf Dependence aware tests for ppc_*_ecdf Apr 5, 2026
@florence-bockting florence-bockting changed the title Dependence aware tests for ppc_*_ecdf Dependence aware tests for ppc*ecdf Apr 5, 2026
@florence-bockting
Copy link
Copy Markdown
Author

As mentioned in the PR introduction text, the idea would be migration in multiple stages.
First stage is non-breaking behavior: That is users can simply use "original" code but are informed that this behavior will change in future. Future changes consider changing the default method and thus are breaking.

Informing the user in the first stage is currently implemented as follows:

# using the "original" code provides a message about the change
> ppc_loo_pit_ecdf(y, yrep, lw)
ℹ In the next major release, the default `method` will change to 'correlated'.
• To silence this message, explicitly set `method = 'independent'` or `method = 'correlated'`.

# explicit use of "independent" method provides information that this method is superseded
> ppc_loo_pit_ecdf(y, yrep, lw, method = "independent")
The 'independent' method is superseded by the 'correlated' method.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants