Add plot with ESS against KS result for TMB and PCA-AGHQ #55

Author: athowes

src/docs_paper/appendix.Rmd

@@ -682,6 +682,10 @@ knitr::include_graphics("depends/rho_a.png")
 knitr::include_graphics("depends/alpha_x.png")
 ```
 
+```{r ks-ess, fig.cap="The ESS was negatively correlated with KS test statistic for both TMB and PCA-AGHQ. This could be explained in two ways. First, harder to sample parameters could be more difficult to estimate with approximate methods. Second, as more effective samples are collected, the NUTS posterior could become closer to the approximate posteriors."}
+knitr::include_graphics("depends/ks-ess.png")
+```
+
 \clearpage
 
 # Algorithm using Laplace latent field marginals\label{sec:laplace-marginals}

src/docs_paper/orderly.yml

@@ -47,6 +47,7 @@ depends:
         depends/ks-summary.rds: ks-summary.rds
         depends/time-taken.csv: time-taken.csv
         depends/beta_alpha.rds: beta_alpha.rds
+        depends/ks-ess.png: ks-ess.png
   - plot-tikz_algorithm-flowchart:
       id: latest
       use:

src/docs_paper/paper.Rmd

@@ -540,7 +540,8 @@ knitr::include_graphics("depends/mean-sd-alt-output.png")
 ### Distribution tests
 
 The two-sample Kolmogorov-Smirnov (KS) test statistic [@smirnov1948table] is the maximum absolute difference between two ECDFs $F(\omega) = \frac{1}{n} \sum_{i = 1}^n \mathbb{I}_{\psi_i \leq \omega}$.
-See an illustration of the KS test in Figure \ref{fig:ks} and a summary of the results in Table.
+See Figure \ref{fig:ks} for an example.
+Additionally, we found that KS test results were negatively correlated with ESS (Figure \ref{fig:ks-ess}).
 
 ```{r ks, fig.cap="Example KS test for one parameter."}
 knitr::include_graphics("figC.png")

src/naomi-simple_ks/ks.Rmd

@@ -560,18 +560,28 @@ Is there any correlation between the value of $\text{KS}(\texttt{method}, \textt
 ```{r}
 rhats <- bayesplot::rhat(tmbstan$mcmc$stanfit)
 ess_ratio <- bayesplot::neff_ratio(tmbstan$mcmc$stanfit)
-niter <- 4 * tmbstan_details$niter / tmbstan_details$nthin
+niter <- 0.5 * 4 * tmbstan_details$niter / tmbstan_details$nthin
 ess <- ess_ratio * niter
 
+fct_reorg <- function(fac, ...) {
+  fct_recode(fct_relevel(fac, ...), ...)
+}
+
 ks_df %>%
   filter(type != "Hyper") %>%
+  mutate(method = fct_reorg(method, "TMB" = "TMB", "PCA-AGHQ" = "aghq")) %>%
   left_join(data.frame(ess) %>%
     tibble::rownames_to_column("par"),
   ) %>%
   ggplot(aes(x = ess, y = ks)) +
-    geom_point(alpha = 0.5) +
-    geom_smooth(method = "lm", color = "#56B4E9") +
+    geom_point(shape = 1, alpha = 0.5) +
+    geom_smooth(method = "lm", color = "#56B4E9", fullrange = TRUE) +
+    stat_poly_eq(use_label("eq"), label.x.npc = "right") +
+    stat_poly_eq(label.y = 0.9, label.x.npc = "right") +
+    scale_x_continuous(limits = c(0, NA)) +
     facet_grid(~ method) +
     theme_minimal() +
-    labs(x = "ESS", y = "KS(method, tmbstan)")
+    labs(x = "ESS", y = "KS(method, NUTS)")
+
+ggsave("ks-ess.png", h = 4, w = 6.25)
 ```

src/naomi-simple_ks/orderly.yml

@@ -21,6 +21,10 @@ artefacts:
       description: KS test summary output
       filenames:
         - ks-summary.rds
+  - data:
+      description: Saved plots
+      filenames:
+        - ks-ess.png
 
 resources:
   - ks.Rmd