Templates for #41 and #42

docs/README.md

@@ -29,6 +29,8 @@
 ### Results
 
 * [ks](https://athowes.github.io/elgm-inf/ks.html): comparison of inference methods for the simplified Naomi model using histograms and KS test results
+* [psis](https://athowes.github.io/elgm-inf/psis.html): comparison of inference methods for the simplified Naomi model using Pareto-smoothed importance sampling
+* [mmd](https://athowes.github.io/elgm-inf/mmd.html): comparison of inference methods for the simplified Naomi model using maximum mean discrepancy
 * [mcmc-convergence](https://athowes.github.io/elgm-inf/mcmc-convergence.html): assessing MCMC (NUTS using `tmbstan`) convergence for the simplified Naomi model
 * [model-checks](https://athowes.github.io/elgm-inf/model-checks.html): checking the fit of the simplified Naomi model to data
 

make/_make.R

@@ -83,6 +83,8 @@ orderly::orderly_commit(id)
 #' Results
 run_commit_push("naomi-simple_mcmc")
 run_commit_push("naomi-simple_ks")
+run_commit_push("naomi-simple_psis")
+run_commit_push("naomi-simple_mmd")
 run_commit_push("naomi-simple_model-checks")
 
 #' Checks

src/naomi-simple_mmd/mmd.Rmd

@@ -0,0 +1,84 @@
+---
+title: "Inference methods comparison for the simplified Naomi model using maximum mean discrepancy"
+author:
+- name: Adam Howes
+output:
+  bookdown::html_document2:
+    toc: yes
+    toc_float: yes
+    toc_collapsed: yes
+    df_print: paged
+    code_folding: show
+    theme: lumen
+abstract: |
+  **Background**  We have run the simplified Naomi model using a range of inference methods: `TMB`, `aghq`, `adam` and `tmbstan`.
+  
+  **Task** In this report, we compare the accuracy of the posterior distributions obtained from these inference methods using maximum mean discrepancy.
+---
+
+# Background
+
+We compare the inference results from `TMB`, `aghq`, `adam`, and `tmbstan`.
+Import these inference results as follows:
+
+```{r}
+tmb <- readRDS("depends/tmb.rds")
+aghq <- readRDS("depends/aghq.rds")
+adam <- readRDS("depends/adam.rds")
+adam_time <- adam$time
+adam <- adam$adam
+tmbstan <- readRDS("depends/tmbstan.rds")
+
+depends <- yaml::read_yaml("orderly.yml")$depends
+```
+
+Check that the parameters (latent field, hyperparameters, model outputs) sampled from each of the four methods are the same:
+
+```{r}
+stopifnot(names(tmb$fit$sample) == names(aghq$quad$sample))
+stopifnot(names(tmb$fit$sample) == names(adam$adam$sample))
+stopifnot(names(tmb$fit$sample) == names(tmbstan$mcmc$sample))
+```
+
+## Run details {.tabset}
+
+For more information about the conditions under which these results were generated, see:
+
+### `TMB`
+
+```{r}
+dependency_details <- function(i) {
+  report_name <- names(depends[[i]])
+  print(paste0("Inference results obtained from ", report_name, " with the query ", depends[[i]][[report_name]]$id))
+  report_id <- orderly::orderly_search(query = depends[[i]][[report_name]]$id, report_name)
+  print(paste0("Obtained report had ID ", report_id, " and was run with the following parameters:"))
+  print(orderly::orderly_info(report_id, report_name)$parameters)
+}
+
+dependency_details(1)
+```
+
+### `aghq`
+
+```{r}
+dependency_details(2)
+```
+
+### `adam`
+
+```{r}
+dependency_details(3)
+```
+
+### `tmbstan`
+
+```{r}
+tmbstan_details <- dependency_details(4)
+tmbstan_details
+```
+
+# Maximum mean discrepancy
+
+```{r}
+#' To write!
+```

src/naomi-simple_mmd/orderly.yml

@@ -0,0 +1,45 @@
+script: script.R
+
+sources:
+  - functions.R
+
+artefacts:
+  - data:
+      description: Notebook comparing inferences
+      filenames:
+        - mmd.html
+
+resources:
+  - mmd.Rmd
+
+packages:
+  - dplyr
+  - forcats
+  - ggplot2
+  - readr
+  - tidyr
+  - purrr
+  - stringr
+  - Matrix
+  - rstan
+  - bayesplot
+  - patchwork
+  - gt
+
+depends:
+  - naomi-simple_fit:
+      id: latest(parameter:tmb == TRUE && parameter:random_only == TRUE)
+      use:
+        depends/tmb.rds: out.rds
+  - naomi-simple_fit:
+      id: latest(parameter:aghq == TRUE && parameter:k == 3 && parameter:s == 8)
+      use:
+        depends/aghq.rds: out.rds
+  - naomi-simple_fit:
+      id: latest(parameter:adam == TRUE)
+      use:
+        depends/adam.rds: out.rds
+  - naomi-simple_fit:
+      id: latest(parameter:tmbstan == TRUE && parameter:niter > 50000)
+      use:
+        depends/tmbstan.rds: out.rds

src/naomi-simple_mmd/script.R

@@ -0,0 +1,5 @@
+#' Uncomment and run the two line below to resume development of this script
+# orderly::orderly_develop_start("naomi-simple_mmd")
+# setwd("src/naomi-simple_mmd")
+
+rmarkdown::render("mmd.Rmd")

src/naomi-simple_psis/orderly.yml

@@ -0,0 +1,45 @@
+script: script.R
+
+sources:
+  - functions.R
+
+artefacts:
+  - data:
+      description: Notebook comparing inferences
+      filenames:
+        - psis.html
+
+resources:
+  - psis.Rmd
+
+packages:
+  - dplyr
+  - forcats
+  - ggplot2
+  - readr
+  - tidyr
+  - purrr
+  - stringr
+  - Matrix
+  - rstan
+  - bayesplot
+  - patchwork
+  - gt
+
+depends:
+  - naomi-simple_fit:
+      id: latest(parameter:tmb == TRUE && parameter:random_only == TRUE)
+      use:
+        depends/tmb.rds: out.rds
+  - naomi-simple_fit:
+      id: latest(parameter:aghq == TRUE && parameter:k == 3 && parameter:s == 8)
+      use:
+        depends/aghq.rds: out.rds
+  - naomi-simple_fit:
+      id: latest(parameter:adam == TRUE)
+      use:
+        depends/adam.rds: out.rds
+  - naomi-simple_fit:
+      id: latest(parameter:tmbstan == TRUE && parameter:niter > 50000)
+      use:
+        depends/tmbstan.rds: out.rds

src/naomi-simple_psis/psis.Rmd

@@ -0,0 +1,89 @@
+---
+title: "Inference methods comparison for the simplified Naomi model using Pareto-smoothed importance sampling"
+author:
+- name: Adam Howes
+output:
+  bookdown::html_document2:
+    toc: yes
+    toc_float: yes
+    toc_collapsed: yes
+    df_print: paged
+    code_folding: show
+    theme: lumen
+abstract: |
+  **Background**  We have run the simplified Naomi model using a range of inference methods: `TMB`, `aghq`, `adam` and `tmbstan`.
+  
+  **Task** In this report, we compare the accuracy of the posterior distributions obtained from these inference methods using Pareto-smoothed importance sampling.
+---
+
+# Background
+
+We compare the inference results from `TMB`, `aghq`, `adam`, and `tmbstan`.
+Import these inference results as follows:
+
+```{r}
+tmb <- readRDS("depends/tmb.rds")
+aghq <- readRDS("depends/aghq.rds")
+adam <- readRDS("depends/adam.rds")
+adam_time <- adam$time
+adam <- adam$adam
+tmbstan <- readRDS("depends/tmbstan.rds")
+
+depends <- yaml::read_yaml("orderly.yml")$depends
+```
+
+Check that the parameters (latent field, hyperparameters, model outputs) sampled from each of the four methods are the same:
+
+```{r}
+stopifnot(names(tmb$fit$sample) == names(aghq$quad$sample))
+stopifnot(names(tmb$fit$sample) == names(adam$adam$sample))
+stopifnot(names(tmb$fit$sample) == names(tmbstan$mcmc$sample))
+```
+
+## Run details {.tabset}
+
+For more information about the conditions under which these results were generated, see:
+
+### `TMB`
+
+```{r}
+dependency_details <- function(i) {
+  report_name <- names(depends[[i]])
+  print(paste0("Inference results obtained from ", report_name, " with the query ", depends[[i]][[report_name]]$id))
+  report_id <- orderly::orderly_search(query = depends[[i]][[report_name]]$id, report_name)
+  print(paste0("Obtained report had ID ", report_id, " and was run with the following parameters:"))
+  print(orderly::orderly_info(report_id, report_name)$parameters)
+}
+
+dependency_details(1)
+```
+
+### `aghq`
+
+```{r}
+dependency_details(2)
+```
+
+### `adam`
+
+```{r}
+dependency_details(3)
+```
+
+### `tmbstan`
+
+```{r}
+tmbstan_details <- dependency_details(4)
+tmbstan_details
+```
+
+# Pareto-smoothed importance sampling
+
+Suppose we have two sets of samples from the posterior.
+For each sample we are going to want to evaluate the log-likelihood, so that we can calculate the log-likelihood ratios.
+We can extract the `TMB` objective function for the log-likelihood as follows:
+
+```{r}
+tmb$fit$obj$fn()
+#' To write!
+```

src/naomi-simple_psis/script.R

@@ -0,0 +1,5 @@
+#' Uncomment and run the two line below to resume development of this script
+# orderly::orderly_develop_start("naomi-simple_psis")
+# setwd("src/naomi-simple_psis")
+
+rmarkdown::render("psis.Rmd")

utils/archive_to_docs.R

@@ -22,6 +22,7 @@ archive_to_docs("docs_01-07-21-stats-epi-group")
 archive_to_docs("docs_15-11-22-seminar")
 archive_to_docs("docs_18-04-23-explainer")
 archive_to_docs("docs_18-04-23-lab-group")
+archive_to_docs("docs_19-05-23-turing")
 archive_to_docs("docs_21-06-23-mlgh")
 archive_to_docs("epil")
 archive_to_docs("prev-anc-art_results")
@@ -39,5 +40,7 @@ archive_to_docs("docs_bioinference-poster")
 archive_to_docs("plot-tikz_algorithm-flowchart")
 archive_to_docs("naomi-simple_mcmc")
 archive_to_docs("naomi-simple_ks")
+archive_to_docs("naomi-simple_psis")
+archive_to_docs("naomi-simple_mmd")
 archive_to_docs("naomi-simple_model-checks")
 archive_to_docs("naomi-simple_increase-s-k")