manuscript.Rmd

---
title             : "Cross-national analyses require additional controls to account for the non-independence of nations"
shorttitle        : "Non-independence of nations"

author: 
  - name          : "Scott Claessens"
    affiliation   : "1*"
  - name          : "Thanos Kyritsis"
    affiliation   : "1"
  - name          : "Quentin D. Atkinson"
    affiliation   : "1,2*"

affiliation:
  - id            : "1"
    institution   : "\\footnotesize School of Psychology, University of Auckland, Auckland, New Zealand"
  - id            : "2"
    institution   : "\\footnotesize School of Anthropology and Museum Ethnography, University of Oxford, Oxford, United Kingdom \\newline *email: scott.claessens@gmail.com; q.atkinson@auckland.ac.nz"

abstract: |
 Cross-national analyses test hypotheses about the drivers of variation in national outcomes. However, since nations are connected in various ways, such as via spatial proximity and shared cultural ancestry, cross-national analyses often violate assumptions of non-independence, inflating false positive rates. Here, we show that, despite being recognised as an important statistical pitfall for over 200 years, cross-national research in economics and psychology still does not sufficiently account for non-independence. In a review of the 100 highest-cited cross-national studies of economic development and values, we find that controls for non-independence are rare. When studies do control for non-independence, our simulations suggest that most commonly used methods are insufficient for reducing false positives in non-independent data. In reanalyses of twelve previous cross-national correlations, half of the estimates are compatible with no association after controlling for non-independence using global proximity matrices. We urge social scientists to sufficiently control for non-independence in cross-national research.

keywords          : "spatial non-independence; cultural non-independence; cross-national analyses; simulations; replications; reanalyses"
wordcount         : "5102 words excl. Methods (Methods: 2163 words)"
bibliography      : ["r-references.bib"]
floatsintext      : yes
figurelist        : no
figsintext        : yes
tablelist         : no
footnotelist      : no
linenumbers       : yes
mask              : no
draft             : no
numbersections    : false
documentclass     : "apa6"
classoption       : "man"
output            : papaja::apa6_pdf
header-includes: 
  - \raggedbottom
  - \usepackage{setspace}
  - \AtBeginEnvironment{tabular}{\singlespacing}
  - \AtBeginEnvironment{lltable}{\singlespacing}
  - \AtBeginEnvironment{tablenotes}{\doublespacing}
  - \captionsetup[table]{font={stretch=1.5}}
  - \captionsetup[figure]{font={stretch=1,small}}
  - \nolinenumbers
csl               : nature.csl
urlcolor          : blue
---

\linenumbers

```{r setup, include=FALSE}
knitr::opts_chunk$set(warning = FALSE, message = FALSE, echo = FALSE)
```

# Introduction

Nations are an important population unit structuring modern human life. The nation in which someone is born has a large effect on what they can expect out of life, including their income level[@Caselli2005], life expectancy[@Austin2012], mental health[@Rai2013], subjective well-being[@Diener2009], and educational attainment[@Kirkcaldy2004]. Nations are also among the most important determinants of human cultural variation, with greater cultural similarity within than outside national borders[@White2021].

Given the importance of nations for structuring human behaviour, there is justifiably huge interest in statistical analyses that attempt to predict variation in national outcomes of all kinds. At the time of writing, a search in Web of Science for the term "cross-national" in titles or abstracts returned over 13,000 unique hits. The standard practice for cross-national analyses is to conduct bivariate correlations or multiple regressions with individual data points representing different nations. Such analyses widen the scope of social science beyond Western populations[@Henrich2010;@Pollet2014] and have been used to study, among other topics, the causes of variation in the economic wealth of nations[@Benhabib1994;@Comin2010;@LaPorta1997;@Sachs2001], global patternings of cultural norms and values[@Fincher2008;@Gelfand2011;@Hofstede2001;@Inglehart2000], and the universality and diversity of human behaviour and psychology around the world[@Awad2018;@Rhoads2021;@Schulz2019;@Thomson2018].

However, cross-national analyses are complicated by the fact that nations are not statistically independent data points. Unlike independent random samples from a population, nations are related to one another in a number of ways. First, nations that are closer to one another tend to be more similar than distant nations. This phenomenon is known as spatial non-independence[@Kissling2008], and it occurs because nations in close spatial proximity share characteristics due to local cultural diffusion of ideas[@Hewlett2002] and regional variation in climate and environment[@Kissling2008]. For example, the neighbouring African nations Zambia and Tanzania have similar levels of terrain ruggedness, which has been used to partially explain their similar levels of economic development[@Nunn2012]. This pattern conforms to Tobler's first law of geography: "everything is related to everything else, but near things are more related than distant things"[@Tobler1970] (p. 236).

Second, nations with shared cultural ancestry tend to be more similar than culturally unrelated nations. This is known as cultural phylogenetic non-independence[@Tylor1889;@Naroll1961;@Naroll1965], and occurs because related nations share cultural traits inherited via descent from a common ancestor. Shared cultural ancestry can result in a form of pseudoreplication, whereby multiple instances of the same trait across nations are merely duplicates of the ancestral original[@Tylor1889;@Murdock1969]. For example, the related island nations Tonga and Tuvalu share similar languages and customs due to cultural inheritance from a common Polynesian population dating back more than 1,000 years[@Gray2009;@Green1999]. More recently, shared ancestry explains similarities in institutions, norms, technologies, and values between colonial settlements and their colonisers (e.g. English, French, Spanish, and Portuguese settlements of the Americas). Importantly, these deep cultural connections between nations often span large geographic distances around the world. Tonga and Tuvalu share cultural traits despite being separated by over 1,500 kilometres of ocean, and South American and European nations remain culturally similar today despite their locations on two separate continents. Shared cultural ancestry must therefore be considered independently of spatial proximity in the study of nations.

Spatial and cultural phylogenetic non-independence between nations make cross-national inference challenging. A fundamental assumption of regression analysis is that model residuals should be independently and identically distributed[@Jarque1987]. But without accounting for spatial or cultural non-independence between nations, model residuals can show structure that remains unaccounted for, violating this assumption. Treating nations as independent can thus inflate false positive rates[@Legendre1993], producing spurious relationships between variables that in fact only indirectly covary due to spatial or cultural connections[@Roberts2013] (see Supplementary Figure \@ref(fig:dag) for an example causal model).

Non-independence is widely acknowledged in fields that routinely deal with spatially or culturally structured data. In ecology and sociology, studies with regional-level data use a variety of autoregressive models designed to account for spatial patternings[@Lichstein2002;@Loftin1983]. In anthropology, researchers have recognised cultural non-independence as an important statistical pitfall for over 200 years, with issues of cultural pseudoreplication being identified in early comparative studies of marriage practices across societies[@Tylor1889]. In the twentieth century, anthropologists began to emphasise that human societies do not develop independently, but rather exist in a globally interconnected world system linked by shared history and cultural ancestry [@Ferguson1997]. In order to minimise the confounding effects of this non-independence in comparisons of human societies, researchers compiled the Standard Cross-Cultural Sample of 186 cultures which deliberately avoids sampling closely related cultures[@Murdock1969], though it is difficult to completely remove spatial and cultural dependencies[@Dow2008;@Eff2004] and the smaller sample of cultures reduces statistical power. Today, anthropologists borrow phylogenetic comparative methods from evolutionary biology, such as phylogenetic least squares regression[@Symonds2014], when comparing societies, treating culturally related societies in the same way as biologists treat genetically related species (e.g. [@Watts2018;@Atkinson2016]).

At the national level, recent reanalyses have revealed that several cross-national relationships reported in economics and psychology do not hold when controlling for non-independence between nations. One working paper replicated 25 analyses of persistence in economics, in which modern national outcomes are regressed against historical characteristics of those nations, and found that over half of the relationships were attenuated when controlling for spatial non-independence[@Kelly2020]. Another replication study found that many of the widely publicised relationships between national-level pathogen prevalence and political institutions and attitudes fail to hold when controlling for various kinds of non-independence[@Bromham2018]. These reanalyses, and others[@Bromham2021;@Currie2012;@Passmore2022], raise the question: how widespread a concern is non-independence in studies of national-level outcomes?

To address this question, we consider national-level variables of general interest across the social sciences: economic development and cultural values. These variables are frequently included as both outcomes and predictors in cross-national studies in economics and psychology[@Benhabib1994;@Comin2010;@Fincher2008;@Gelfand2011;@Hofstede2001;@Inglehart2000;@LaPorta1997;@Sachs2001]. First, we demonstrate that economic development and cultural values are spatially and culturally non-independent across nations, emphasising the need to control for non-independence. Second, we review the 100 highest-cited cross-national studies of economic development and cultural values and determine baseline rates of controlling for non-independence in the literature. Third, we run simulations to determine whether common methods of dealing with non-independence in the literature sufficiently reduce false positive rates. Fourth, we reanalyse twelve previous cross-national analyses of economic development and cultural values from our literature review, incorporating global geographic and linguistic proximity matrices as controls for spatial and cultural non-independence.

# Results

## National-level economic development and cultural values are spatially and culturally non-independent

In order to motivate our research question, it is important to first quantify the degree of spatial and cultural non-independence for economic development and cultural values around the world. If these variables are independent or only weakly non-independent, then the issue might be safe to ignore. However, if they are more strongly non-independent, then there is a possibility that non-independence could be confounding cross-national inferences.

To this end, we used Bayesian multilevel models to simultaneously estimate geographic and cultural phylogenetic signal for a range of economic development and cultural values variables. For economic development, we focused on the Human Development Index[@hdi], gross domestic product per capita, annual gross domestic product per capita growth, and the Gini index of income inequality. For cultural values, we focused on two primary dimensions of cultural values from the World Values Survey, traditional vs. secular values and survival vs. self-expression values[@Inglehart2000], as well as cultural tightness[@Gelfand2011] and individualism[@Hofstede2001]. These variables are not intended to be a comprehensive list of all national-level variables included in cross-national research, but rather an illustrative set of variables that are widely used in the literature.

For all of these variables, we found that a substantial proportion of national-level variation was explained by spatial proximity and/or shared cultural ancestry between nations (Figure \@ref(fig:plotSignal); see Supplementary Table \@ref(tab:tableSignal) for numerical results). Signal estimates were often strong, with spatial proximity and shared cultural ancestry frequently explaining over half of the national-level variation. For spatial proximity, Bayes Factors indicated strong evidence that the geographic signal estimates differed from zero for all economic development variables and traditional values. However, the evidence was only equivocal for survival values and individualism, and strong evidence was found that the geographic signal estimate for tightness was equal to zero. For shared cultural ancestry, Bayes Factors indicated strong evidence that the cultural phylogenetic signal estimates differed from zero for all economic development and cultural values variables except for gross domestic product per capita growth, for which the evidence was equivocal. These findings emphasise the need to account for spatial and cultural phylogenetic non-independence in cross-national analyses of economic development and cultural values.

(ref:plotSignalCaption) Posterior estimates of geographic and cultural phylogenetic signal for a range of economic development and cultural values variables. Geographic and cultural phylogenetic signal are operationalised as the proportion of national-level variance explained by geographic and linguistic proximity matrices. Grey ridges are full posterior distributions, points are posterior median values, and black lines are equal-tailed 50% and 95% credible intervals. Number of observations from these models are as follows: HDI (n = 1449), GDP per capita (n = 10289), GDP per capita growth (n = 9755), Gini index (n = 1826), traditional values (n = 277676), survival values (n = 277676), tightness (n = 57), and individualism (n = 67). HDI = Human Development Index; GDP = gross domestic product.

```{r plotSignal, fig.width=7, fig.height=5, fig.cap="(ref:plotSignalCaption)"}
tar_read(plotSignal)
```

## Previous cross-national analyses have not sufficiently accounted for non-independence

```{r load review}
tar_load(review)
```

Given that economic development and cultural values show evidence of geographic and cultural phylogenetic signal, have cross-national analyses sufficiently accounted for this non-independence? To assess this, we searched the published literature for articles that combined the search terms "economic development" or "values" with the search terms "cross-national", "cross-cultural", or "cross-country". We removed articles that did not report original research, were not relevant to economic development or cultural values, or did not report at least one cross-national analysis. We then retained the 100 articles (50 for economic development, 50 for cultural values) with the highest annual rate of citations (see Supplementary Data 1). For each of these highly-cited articles, we exhaustively recorded every cross-national analysis reported in the main text (n = `r printnum(as.numeric(nrow(review)), digits = 0)`), identifying in each case whether or not the analysis attempted to control for spatial, cultural, or any other form of non-independence between nations (see Methods for detailed search criteria and coding decisions).

```{r calculate prop}
# article-level
prop <-
  review %>%
  group_by(Review, Citation) %>%
  summarise(Yes = ifelse(sum(ControlNI == "Yes") > 0, 1, 0), .groups = "drop") %>%
  group_by(Review) %>%
  summarise(prop = mean(Yes), .groups = "drop") %>%
  # bootstrap cis
  mutate(lower = map(prop, function(x) getBootCIProp(x, 50, 1000)[1]), 
         upper = map(prop, function(x) getBootCIProp(x, 50, 1000)[2])) %>%
  unnest(c(lower, upper))
# analysis-level
tar_load(postRM1)
propEcon <- inv_logit_scaled(postRM1[,,"b_ReviewEconomicdevelopment"])
propVal  <- inv_logit_scaled(postRM1[,,"b_ReviewValues"])
```

The results of our literature review show that most published articles containing cross-national analyses make no attempt to account for statistical non-independence. Figure \@ref(fig:plotReview1)a plots the proportion of articles that contain at least one cross-national analysis accounting for non-independence. We find that `r printnum(prop$prop[1]*100, digits = 0)`% of economic development articles contain at least one attempt to control for non-independence (two-tailed 95% bootstrap confidence interval [`r printnum(prop$lower[1])` `r printnum(prop$upper[1])`]), while this proportion decreases to only `r printnum(prop$prop[2]*100, digits = 0)`% for cultural values articles (two-tailed 95% bCI [`r printnum(prop$lower[2])` `r printnum(prop$upper[2])`]). Both kinds of article are most likely to use regional fixed effects (e.g. continent fixed effects) to account for non-independence, but some articles also include controls for spatial distance (e.g. latitude) and shared cultural history (e.g. colony status). These proportions are even lower when focusing on the full sample of `r printnum(as.numeric(nrow(review)), digits = 0)` analyses: only `r printnum(median(propEcon)*100, digits = 0)`% (equal-tailed 95% credible interval [`r printnum(quantile(propEcon, 0.025))` `r printnum(quantile(propEcon, 0.975))`]) of individual economic development analyses and `r printnum(median(propVal)*100, digits = 0)`% (equal-tailed 95% CI [`r printnum(quantile(propVal, 0.025))` `r printnum(quantile(propVal, 0.975))`]) of individual cultural values analyses are estimated to control for non-independence (Supplementary Figure \@ref(fig:plotReview2)).

(ref:plotReview1Caption) Results from literature review of 100 highly-cited cross-national studies of economic development (red) and cultural values (blue). (a) Proportion of articles containing at least one analysis accounting for non-independence, overall and split by common methods of controlling for non-independence. Points represent raw proportions of articles and ranges represent two-tailed 95% bootstrap confidence intervals (n = 1000 bootstrap samples). (b) The association between journal impact factor and the probability that an article contains at least one analysis accounting for non-independence. (c) Estimated trend over time for the probability that an article contains at least one analysis accounting for non-independence. Lines and shaded areas are posterior median regression lines and equal-tailed 50% credible intervals from Bayesian logistic regression models (n = 100 observations). Dots represent raw counts of individual articles that did (top) or did not (bottom) account for non-independence. Region FEs = region fixed effects.

```{r plotReview1, fig.width=7, fig.height=6, fig.cap="(ref:plotReview1Caption)"}
tar_read(plotReview1)
```

```{r load article models, echo=F}
tar_load(yearArticle)
tar_load(ifArticle)
```

While our review contains articles from journals with a range of impact factors, our estimates could be biased downwards by analyses published in lower impact outlets with more relaxed standards for issues like non-independence. It is also possible that, since our literature review goes back as far as 1993, our estimates are being biased downwards by earlier studies, and that controls for non-independence have increased over time with methodological advancements and greater awareness of the issue. To test these possible explanations for our low estimates, we fitted logistic regression models to the data from the review, including log journal impact factor and publication year as separate predictors. Interestingly, we found that, if anything, studies from higher impact journals were less likely to include at least one control for non-independence than studies from lower impact journals, both for studies of economic development (b = `r printnum(fixef(ifArticle)["ReviewEconomicdevelopment:logIF","Estimate"])`, equal-tailed 95% CI [`r printnum(fixef(ifArticle)["ReviewEconomicdevelopment:logIF","Q2.5"])` `r printnum(fixef(ifArticle)["ReviewEconomicdevelopment:logIF","Q97.5"])`]) and for studies of cultural values (b = `r printnum(fixef(ifArticle)["ReviewValues:logIF","Estimate"])`, equal-tailed 95% CI [`r printnum(fixef(ifArticle)["ReviewValues:logIF","Q2.5"])` `r printnum(fixef(ifArticle)["ReviewValues:logIF","Q97.5"])`]; Figure \@ref(fig:plotReview1)b). Moreover, splines revealed no relationship between publication year and the probability of including at least one control for non-independence, both for studies of economic development (b = `r printnum(fixef(yearArticle)["sYear.std:ReviewEconomicdevelopment_1","Estimate"])`, equal-tailed 95% CI [`r printnum(fixef(yearArticle)["sYear.std:ReviewEconomicdevelopment_1","Q2.5"])` `r printnum(fixef(yearArticle)["sYear.std:ReviewEconomicdevelopment_1","Q97.5"])`]) and for studies of cultural values (b = `r printnum(fixef(yearArticle)["sYear.std:ReviewValues_1","Estimate"])`, equal-tailed 95% CI [`r printnum(fixef(yearArticle)["sYear.std:ReviewValues_1","Q2.5"])` `r printnum(fixef(yearArticle)["sYear.std:ReviewValues_1","Q97.5"])`]; Figure \@ref(fig:plotReview1)c).

## Common methods of controlling for non-independence are insufficient for reducing false positive rates in non-independent data

Our literature review revealed that most cross-national analyses in the literature do not control for spatial or cultural phylogenetic non-independence. When they do, they tend to include controls like latitude and regional fixed effects. Do these methods sufficiently account for statistical non-independence?

To compare the efficacy of different methods in the literature, we conducted a simulation study. We simulated national-level datasets (n = 236 nations) with varying degrees of spatial or cultural phylogenetic autocorrelation (i.e. non-independence) for outcome and predictor variables, but with no direct causal relationship between the variables. We then fitted naive regressions without controls to these datasets, as well as regression models with controls for latitude, longitude, and continent fixed effects. Despite not being identified in our literature review, we also included other methods that are often used in the literature to account for non-independence. Additional spatial controls included the mean of the predictor variable within a surrounding 2000km radius (e.g. [@Schulz2022]) and Conley standard errors[@Conley1999;@Conley2010] based on geographic distances between nations (e.g. [@Schulz2019;@Schulz2022]). Additional cultural controls included fixed effects for the language families of the majority-spoken languages in each nation (e.g. [@Currie2009]) and Conley standard errors based on genetic distances between nations (e.g. [@Schulz2019;@Schulz2022]). These fixed effect approaches attempt to account for non-independence by holding geographic location constant (latitude, longitude), discarding between-region variation and exploiting only local variation (continent fixed effects, mean of surrounding 2000km), or correcting standard errors for autocorrelation post-hoc while leaving model coefficients unchanged (Conley standard errors).

Beyond fixed effect approaches, we also fitted Bayesian random effects regressions that explicitly model spatial and/or cultural phylogenetic non-independence by allowing nations to covary according to geographic and/or linguistic proximity matrices. Geographic proximity between nations is calculated from inverse distances between longitude and latitude coordinates. Linguistic proximity between nations is calculated from a global phylogenetic tree that represents hierarchical relationships of genealogical descent for all languages in the world. For each pair of nations, we calculated inverse phylogenetic distances (i.e. number of branches separating two taxa) between all languages spoken in that nation pair and produced an average linguistic proximity score weighted by the percentages of speakers within those nations. To include the resulting geographic and linguistic proximity matrices in our models, we included a Gaussian process[@McElreath2020;@Neal1998] over latitude and longitude values and/or assumed that nation random intercepts were correlated in proportion to their linguistic proximity[@deVillemereuil2014]. These random effects approaches attempt to account for non-independence by modelling the covariance between nations that is induced by their geographic or linguistic connections.

```{r calculate false positive rate}
fpGeo <-
  tar_read(olsModel1_simGeoCov) %>%
  filter(r == 0) %>%
  group_by(lambda, rho) %>%
  summarise(n = sum(sig) / n(), .groups = "drop")
fpLin <-
  tar_read(olsModel1_simLinCov) %>%
  filter(r == 0) %>%
  group_by(lambda, rho) %>%
  summarise(n = sum(sig) / n(), .groups = "drop")
fpGeoContinent <-
  tar_read(olsModel4_simGeoCov) %>%
  filter(r == 0) %>%
  group_by(lambda, rho) %>%
  summarise(n = sum(sig) / n(), .groups = "drop")
fpGeoRadius <-
  tar_read(olsModel6_simGeoCov) %>%
  filter(r == 0) %>%
  group_by(lambda, rho) %>%
  summarise(n = sum(sig) / n(), .groups = "drop")
fpGeoSpatial <-
  tar_read(brmsModel1_simGeoCov) %>%
  filter(r == 0) %>%
  group_by(lambda, rho) %>%
  summarise(n = sum(sig) / n(), .groups = "drop")
fpLinLangFam <-
  tar_read(olsModel5_simLinCov) %>%
  filter(r == 0) %>%
  group_by(lambda, rho) %>%
  summarise(n = sum(sig) / n(), .groups = "drop")
```

Figures \@ref(fig:plotSim3) and \@ref(fig:plotSim4) plot the estimated false positive rates from our simulation study, split by different methods and different degrees of spatial or cultural phylogenetic autocorrelation (see Supplementary Tables \@ref(tab:tableSim1) and \@ref(tab:tableSim2) for numerical results and Supplementary Figures \@ref(fig:plotSim1) and \@ref(fig:plotSim2) for full distributions of effect sizes under strong autocorrelation). Across all model types, false positive rates were measured as the proportion of models that estimated a slope with a two-tailed 95% confidence / credible interval excluding zero, falsely inferring a relationship when none is present. For reference, weak autocorrelation in our simulation is comparable to the geographic signal for survival values in Figure \@ref(fig:plotSignal) (i.e. 20% of the national-level variance is explained by non-independence), while moderate and strong levels of autocorrelation are comparable to the cultural phylogenetic signal for traditional and survival values, respectively (i.e. 50% and 80% of the national-level variance is explained by non-independence).

Our simulation study revealed that with at least moderate degrees of spatial or cultural phylogenetic autocorrelation for both outcome and predictor variables, naive regression models produce false positive rates above chance levels. This false positive rate increases as the degree of autocorrelation increases. With strong spatial autocorrelation for both outcomes and predictors, false positive rates reach as high as `r printnum(fpGeo$n[fpGeo$lambda == 0.8 & fpGeo$rho == 0.8]*100, digits = 0)`%. We find a slightly lower false positive rate under strong cultural phylogenetic autocorrelation, though this false positive rate is still greater than expected by chance (`r printnum(fpLin$n[fpLin$lambda == 0.8 & fpLin$rho == 0.8]*100, digits = 0)`%).

(ref:plotSim3Caption) False positive rates for different methods of controlling for spatial non-independence in our simulation study. For simulated outcome and predictor variables, we systematically varied the strength of spatial autocorrelation, from weak (0.2) to moderate (0.5) to strong (0.8). We simulated 100 datasets per parameter combination and fitted different models to each dataset. False positive rates were operationalised as the proportion of models that estimated a slope with a two-tailed 95% confidence / credible interval excluding zero. Points represent raw proportions of false positive models, ranges represent two-tailed 95% bootstrap confidence intervals (n = 1000 bootstrap samples), and dashed lines indicate the 5% false positive rate that is expected due to chance. Colours indicate whether the strength of autocorrelation for the predictor variable is 0.2 (red), 0.5, (green) or 0.8 (blue). SEs = standard errors.

```{r plotSim3, fig.width=7, fig.height=5.5, fig.cap="(ref:plotSim3Caption)"}
tar_read(plotSim3)
```

(ref:plotSim4Caption) False positive rates for different methods of controlling for cultural phylogenetic non-independence in our simulation study. For simulated outcome and predictor variables, we systematically varied the strength of cultural phylogenetic autocorrelation, from weak (0.2) to moderate (0.5) to strong (0.8). We simulated 100 datasets per parameter combination and fitted different models to each dataset. False positive rates were operationalised as the proportion of models that estimated a slope with a two-tailed 95% confidence / credible interval excluding zero. Points represent raw proportions of false positive models, ranges represent two-tailed 95% bootstrap confidence intervals (n = 1000 bootstrap samples), and dashed lines indicate the 5% false positive rate that is expected due to chance. Colours indicate whether the strength of autocorrelation for the predictor variable is 0.2 (red), 0.5, (green) or 0.8 (blue). SEs = standard errors.

```{r plotSim4, fig.width=7, fig.height=5.5, fig.cap="(ref:plotSim4Caption)"}
tar_read(plotSim4)
```

Most methods common in the literature do not reduce these high false positive rates. With strong spatial autocorrelation for both outcome and predictor variables, false positive rates remain above 50% when controlling for latitude, longitude, and language family fixed effects (Figure \@ref(fig:plotSim3)). Similarly, Conley standard error corrections based on spatial and genetic distances do not reduce false positive rates below 40% under strong spatial autocorrelation. The most effective fixed effects methods are continent fixed effects, which continue to produce a false positive rate of `r printnum(fpGeoContinent$n[fpGeoContinent$lambda == 0.8 & fpGeoContinent$rho == 0.8]*100, digits = 0)`% under strong spatial autocorrelation, and controlling for the mean of the predictor variable within a 2000km radius, which eliminates false positives under even strong spatial autocorrelation (`r printnum(fpGeoRadius$n[fpGeoRadius$lambda == 0.8 & fpGeoRadius$rho == 0.8]*100, digits = 0)`%). However, additional simulations revealed that these reductions in false positive rates come at the cost of lower statistical power (see Supplementary Figures \@ref(fig:plotPowerGeo01) - \@ref(fig:plotPowerGeo05)). In additional simulations where the true relationship between the predictor and outcome variable was known, power analyses showed that both continent fixed effects and the 2000km radius control had less than 80% power to detect moderate true correlations (r = 0.3) under strong spatial autocorrelation.

By contrast, Bayesian spatial Gaussian process regression with longitude and latitude strikes a balance between reducing false positives and retaining high statistical power to detect true effects. This approach reduces false positives to `r printnum(fpGeoSpatial$n[fpGeoSpatial$lambda == 0.5 & fpGeoSpatial$rho == 0.5]*100, digits = 0)`% under moderate spatial autocorrelation and `r printnum(fpGeoSpatial$n[fpGeoSpatial$lambda == 0.8 & fpGeoSpatial$rho == 0.8]*100, digits = 0)`% under strong spatial autocorrelation. Random effects models that additionally account for linguistic proximity between nations perform equally well, though models with only linguistic covariance continue to produce false positives. False positives are not completely eliminated with these random effects models. Nevertheless, these methods have at least 80% power to detect moderate (r = 0.3) and large (r = 0.5) true correlations between variables under all levels of spatial autocorrelation (see Supplementary Figures \@ref(fig:plotPowerGeo01) - \@ref(fig:plotPowerGeo05)).

In our simulation of cultural phylogenetic non-independence, we find that none of the fixed effects methods reduce false positive rates (Figure \@ref(fig:plotSim4)). Controls for latitude, longitude, continent fixed effects, the mean of the predictor variable in a 2000km radius, and Conley standard error corrections based on spatial and genetic distances do little to change false positive rates. Even language family fixed effects continue to produce a false positive rate of `r printnum(fpLinLangFam$n[fpLinLangFam$lambda == 0.8 & fpLinLangFam$rho == 0.8]*100, digits = 0)`% under strong cultural phylogenetic autocorrelation. By contrast, models with random effects covarying according to linguistic proximity completely eliminate false positives across all degrees of cultural phylogenetic autocorrelation. This approach is also the only method that is able to detect large true correlations (r = 0.5) with at least 80% power (see Supplementary Figures \@ref(fig:plotPowerLin01) - \@ref(fig:plotPowerLin05)). Random effects models that additionally account for geographic proximity between nations perform equally well, though models with only a spatial Gaussian process continue to produce false positives.

## Key findings in the literature are not robust to reanalysis with more rigorous methods

Our literature review and simulation study have shown that controls for non-independence are rare in cross-national studies of economic development and cultural values, and when studies do attempt to control for non-independence, the methods typically used are unable to sufficiently reduce false positive rates. This raises the worrying possibility that the cross-national literature in economics and psychology is populated with spurious relationships.

To determine how widespread this issue of spurious cross-national relationships might be, we reanalysed a subset of twelve previous cross-national analyses from our literature review, controlling for spatial and cultural phylogenetic non-independence using global geographic and linguistic proximity matrices. Out of the 100 papers included in our literature review, primary or secondary data were publicly available for `r printnum(review %>% group_by(Review, Citation) %>% summarise(DataAvailable = ifelse(mean(DataAvailable == "Yes") > 0, TRUE, FALSE), .groups = "drop") %>% pull(DataAvailable) %>% sum())` papers. We attempted to replicate key statistically significant cross-national correlations from these papers --- mostly initial bivariate regression specifications without covariates (see Methods) --- and stopped when we had sampled a set of twelve analyses for which we were able to replicate the original result. The final set included six analyses from our economic development review[@Beck2003;@Beck2005;@Bockstette2002;@Easterly2003;@Easterly2007;@Skidmore2002] and six from our cultural values review[@Adamczyk2009;@Alesina2013;@Fincher2008;@Gelfand2011;@Inglehart2000;@Knack1997] that had available data and were able to be replicated. We pre-registered this set before running any control models (https://osf.io/uywx8/). We controlled for non-independence by including (1) a Gaussian process allowing nation random intercepts to covary according to a geographic proximity matrix, and/or (2) nation random intercepts that covaried according to a linguistic proximity matrix (see Supplementary Methods for full models).

Figure \@ref(fig:plotReplications1) visualises the results of our reanalysis (see Supplementary Table \@ref(tab:tableReplications) for numerical results). Cross-national correlation effect sizes tended to reduce when controlling for statistical non-independence between nations, sometimes by as much as half of the original effect size. Overall, after controlling for non-independence, six out of twelve cross-national associations had 95% credible intervals that included zero. For the economic development analyses, four out of six cross-national relationships had 95% credible intervals including zero when controlling for spatial non-independence. For the cultural values analyses, two out of six cross-national relationships had 95% credible intervals including zero when controlling for cultural phylogenetic non-independence. Supplementary Figure \@ref(fig:plotReplications2) shows these cross-national correlations plotted against the raw data.

(ref:plotReplications1Caption) Posterior correlations from our reanalysis of twelve previous cross-national analyses. For each previous cross-national relationship, we plot the posterior slopes from a naive regression (red), a regression controlling for spatial non-independence (green), a regression controlling for cultural phylogenetic non-independence (blue), and a regression controlling for both spatial and cultural phylogenetic non-independence simultaneously (purple). All outcome and predictor variables are standardised. Most analyses are simple bivariate cross-national correlations, but Gelfand et al. (2011) is a partial correlation controlling for log gross national income and Adamczyk and Pitt (2009) is a multilevel model including several covariates. Coloured ridges are full posterior distributions, and points and black lines represent posterior medians and equal-tailed 95% credible intervals. Numbers of observations from the models are as follows, from top to bottom: n = 69, n = 45, n = 103, n = 98, n = 63, n = 89, n = 33, n = 75, n = 67, n = 28, n = 38, and n = 28. GDP = gross domestic product. FLFP = female labour force participation.

```{r plotReplications1, fig.width=8, fig.height=6.5, fig.cap="(ref:plotReplications1Caption)"}
tar_read(plotReplications1)
```

```{r load autocor}
tar_load(c(inglehartSA1, skidmoreSA1, alesinaSignal1, knackSignal1, effSizeRatioModel1, effSizeRatioModel2))
```

To understand why some cross-national correlations were attenuated by controls for non-independence while others were robust, we further explored our fitted models for evidence of spatial and cultural autocorrelation. For each outcome variable, our Gaussian process models provided varying estimates of how quickly spatial autocorrelation declined with distance (Supplementary Figure \@ref(fig:plotReplications3)). For example, in Skidmore and Toya[@Skidmore2002] gross domestic product growth was only moderately spatially autocorrelated at 1,000 km distance (posterior median spatial autocorrelation at 1,000 km = `r printnum(median(skidmoreSA1))`, equal-tailed 95% CI [`r printnum(quantile(skidmoreSA1, 0.025))` `r printnum(quantile(skidmoreSA1, 0.975))`]), whereas in Inglehart and Baker[@Inglehart2000] traditional values were strongly spatially autocorrelated at the same distance (posterior median spatial autocorrelation at 1,000 km = `r printnum(median(inglehartSA1))`, 95% CI [`r printnum(quantile(inglehartSA1, 0.025))` `r printnum(quantile(inglehartSA1, 0.975))`]). We also found varying estimates of cultural phylogenetic signal (Supplementary Figure \@ref(fig:plotReplications4)), with some outcome variables expressing low signal (e.g. confidence in institutions[@Knack1997]; posterior median = `r printnum(median(knackSignal1))`, 95% CI [`r printnum(quantile(knackSignal1, 0.025))` `r printnum(quantile(knackSignal1, 0.975))`]) and others expressing high signal (e.g. female labour force participation[@Alesina2013]; posterior median = `r printnum(median(alesinaSignal1))`, 95% CI [`r printnum(quantile(alesinaSignal1, 0.025))` `r printnum(quantile(alesinaSignal1, 0.975))`]). Exploratory regressions provided suggestive evidence that stronger estimates of spatial autocorrelation or cultural phylogenetic signal resulted in a more pronounced reduction in the effect size when controlling for non-independence between nations (Supplementary Figure \@ref(fig:plotReplications5)). However, these negative slopes for spatial autocorrelation (b = `r printnum(fixef(effSizeRatioModel1)["corAt1000km","Estimate"])`, 95% CI [`r printnum(fixef(effSizeRatioModel1)["corAt1000km","Q2.5"])` `r printnum(fixef(effSizeRatioModel1)["corAt1000km","Q97.5"])`]) and cultural phylogenetic signal (b = `r printnum(fixef(effSizeRatioModel2)["signal","Estimate"])`, 95% CI [`r printnum(fixef(effSizeRatioModel2)["signal","Q2.5"])` `r printnum(fixef(effSizeRatioModel2)["signal","Q97.5"])`]) were both very uncertain, due to the small number of analyses and the posterior uncertainty in effect sizes and estimates of non-independence.

# Discussion

In a literature review and simulation, we found that cross-national studies in economics and psychology rarely account for non-independence between nations, and, when they do, the methods they use are insufficient to reduce false positives in non-independent data. In a reanalysis of twelve cross-national correlations, we further showed that neglecting to account for non-independence has resulted in potentially spurious relationships in the published literature, with half of the correlations failing to replicate when controlling for spatial or cultural non-independence with more rigorous methods. These findings suggest that cross-national analyses in economics and psychology should be interpreted with caution until non-independence is sufficiently accounted for.

Our initial analyses add to and clarify existing evidence regarding the degree of non-independence for national-level economic and cultural variables. One previous study suggested that geographic proximity is more important than deep cultural ancestry in explaining the distribution of human development across Eurasian nations, though the authors noted that their small sample of 44 nations and regional focus limited their statistical power[@Sookias2018]. By contrast, our global samples of over 160 nations revealed strong cultural phylogenetic signal, as well as geographic signal, for the Human Development Index, GDP per capita, and the Gini index of inequality. Another previous study found that similarities in the cultural values of nations are predicted by linguistic, but not geographic, distances between those nations[@White2021]. We find this same result for survival vs. self-expression values, cultural tightness, and individualism, but for traditional vs. secular values we find that both linguistic and geographic proximity are important independent predictors of global variation. These findings emphasise the need to account for both spatial and cultural phylogenetic non-independence in cross-national studies of economic development and cultural values.

Crucially, our literature review and simulation study revealed that the most commonly used controls for non-independence do not sufficiently deal with the issue. In our simulations, controlling for either latitude or longitude did not reduce false positive rates. This result calls into question the use of controls like distance to the equator to account for non-independence in cross-national regression models, though these controls may still be suitable to account for regional or latitudinal variation in climate, ecology, and natural threats (e.g. pathogens), which we did not simulate. High false positive rates persisted with Conley standard errors, despite recent claims that these standard error corrections are sufficient to deal with spatial non-independence[@Voth2021]. The simulation also confirmed the assertion that fixed effects for spatial or cultural groupings (e.g. continent or language family fixed effects) are insufficient because non-independence still remains within groupings[@Bromham2018]. This logic further applies to analyses that control for non-independence by separately analysing different regions (e.g. [@Smith2017]). Controlling for the mean of the predictor variable within a 2000km radius[@Schulz2022] eliminated false positive rates in spatially autocorrelated data, but had reduced statistical power to detect true associations. Across all model types in our simulation, the only methods that reduced false positive rates while retaining high statistical power were the random effects models with covariance matrices. The important advantage of these models is not that they are Bayesian per se (any of the approaches used here could be implemented in a Bayesian framework), but rather that they explicitly model covariance as a function of spatial or cultural distance.

There are other approaches to controlling for non-independence that we did not include in our simulation. For example, conditional autoregressive models[@Lichstein2002] and generalised additive models[@Hastie2017] are approaches that can be applied in both frequentist and Bayesian frameworks. There are also alternative ways to operationalise cultural distances between nations beyond linguistic distances, including metrics like cultural fixation indices (e.g. cultural $F_{ST}$[@Muthukrishna2020;@White2021]), covariance based on genetic distances[@Spolaore2009], and phylogenetic distances between religious traditions. We see merit in each of these approaches and the use of one over another will depend on the specific question at hand. We decided to focus on linguistic distances in this study since language is a tangible socially-learned trait that has previously been used to successfully track the effects of deep cultural ancestry on modern national outcomes[@Kyritsis2022;@Matthews2016;@Sookias2018]. Future work should explore whether other approaches are sufficient to reduce false positive rates in spatially and culturally non-independent data. In addition to spatial proximity and shared cultural ancestry, we did not simulate other sources of non-independence that potentially exist in real cross-national datasets, such as modern connections between nations due to flows of people and information (e.g. flight networks, social media networks) and shared histories of colonialism and capitalist modes of production that have shaped today's global landscape[@Trouillot2016;@Wolf2010]. Additional controls will be required to ensure that these sources of non-independence do not confound cross-national inferences.

Ours is not the first review to show that studies are misapplying statistical methods in ways that inflate false positive rates. For example, other literature reviews have shown that studies in the social sciences tend to use small samples of participants[@Smaldino2016], treat ordinal data as metric[@Liddell2018], incorrectly handle missing values[@Nicholson2017], and ignore best practices in meta-analyses[@Tipton2019]. Why do cross-national studies also rarely account for non-independence? At the institutional level, one possibility is that such practices are incentivised because they generate statistically significant relationships, which increase the probability that a study is published[@Smaldino2016]. Indeed, we found that controls for non-independence were less common among articles published in high impact journals, suggesting that researchers are rewarded for such practices. At the individual level, another possibility is that researchers outside of anthropology and ecology are less aware of the problem, or believe that the problem does not apply to analyses of nations. Even if researchers appreciate the problem, they might not know of suitable controls or perceive the methods to be too complex.

These institutional- and individual-level barriers can be combatted. First, cross-national replication studies like ours and others[@Bromham2018;@Bromham2021;@Currie2012;@Kelly2020;@Passmore2022], combined with the methodological reviews included in Registered Reports[@Chambers2021], might change incentive structures and encourage researchers to analyse the world's nations with more rigorous methods. Second, more explicit descriptions of causal models could promote controls for non-independence by clearly outlining the nature of confounding and the sources of autocorrelation in cross-national data[@Deffner2022]. The causal model outlined in Supplementary Figure \@ref(fig:dag) is a useful example, but individual studies must outline their own particular causal assumptions, which may include further sources of non-independence and confounding variables to control for (e.g. post-communist status, colony status). These causal models can then be used to design tailored statistical estimation strategies. Indeed, in our review, economists studying economic development dealt with national-level non-independence more than psychologists studying cultural values, likely because economics studies tend to be lengthy statistical exercises that systematically incorporate or exclude numerous variables in an attempt to infer causation. Third, the recent widespread accessibility of open source statistical software, such as the programming language Stan[@rstan] and the R package brms[@Burkner2017], should promote the use of more rigorous methods to control for non-independence. Using brms, for example, Bayesian Gaussian process regression is straightforward to conduct, requiring only longitude and latitude values for nations. We have provided an online tutorial to help researchers apply these methods to their own cross-cultural datasets (https://scottclaessens.github.io/blog/2022/crossnational/).

Until such changes are implemented and sufficient controls for non-independence are the norm, existing cross-national correlations should be interpreted with caution. In our reanalyses, we found that half of the cross-national correlations had equal-tailed 95% credible intervals that included zero when controlling for spatial and/or cultural phylogenetic non-independence. While these results are in line with previous reanalyses[@Kelly2020], we note that we are unable to outright reject the claims from these studies, since we only reanalysed the first bivariate regression specifications presented in the papers[@Voth2021]. More detailed sets of reanalyses would be required to comprehensively challenge the claims from these specific papers. Nevertheless, these reanalyses do show, more broadly, that the problem of statistical non-independence applies to a wider range of national-level variables than those identified by previous work, such as parasite stress and democratic outcomes[@Bromham2018]. Moreover, given our finding that most studies in the current cross-national literature do not deal with non-independence at all, our reanalyses raise the worrying possibility that this literature is populated with spurious relationships. Future work should expand our set of reanalyses to determine the extent of this problem in the literature.

We do not wish to dissuade researchers from conducting cross-national studies. On the contrary, such work promises to deepen understanding of our world, including the causes and consequences of economic development and cultural values. Moreover, cross-national studies allow social scientists to broaden their scope of study beyond Western populations[@Henrich2010], providing the representative samples necessary to test evolutionary and socio-ecological theories of human behaviour[@Pollet2014;@Oishi2013]. But in order to minimise spurious relationships in global datasets, we urge researchers to control for spatial and cultural phylogenetic non-independence when reporting cross-national correlations. Nations are not independent, and our statistical models must reflect this.

# Methods

## Geographic and cultural phylogenetic signal for measures of economic development and cultural values

```{r}
tar_load(c(hdi, gdp, gdpGrowth, gini, wvs, tightness, fincherData))
```

To estimate the degree of spatial and cultural phylogenetic non-independence in economic development and cultural values, we calculated geographic and cultural phylogenetic signal for global measures of development and values. For economic development variables, we retrieved longitudinal data on the Human Development Index[@hdi] (`r min(hdi$year)` - `r max(hdi$year)`; n = `r printnum(length(unique(hdi$Country)))` nations), gross domestic product per capita (`r min(gdp$year)` - `r max(gdp$year)`; n = `r printnum(length(unique(gdp$iso2)))` nations), annual percentage growth in gross domestic product per capita (`r min(gdpGrowth$year)` - `r max(gdpGrowth$year)`; n = `r printnum(length(unique(gdpGrowth$iso2)))` nations), and the Gini coefficient of income inequality (`r min(gini$year)` - `r max(gini$year)`; n = `r printnum(length(unique(gini$iso2)))` nations). Human development data were retrieved from the United Nations Development Programme (https://hdr.undp.org/en/content/download-data) and data for all other economic development variables were retrieved from the World Bank (https://data.worldbank.org/). For cultural values variables, we retrieved longitudinal data on traditional vs. secular values and survival vs. self-expression values from the World Values Survey[@Inglehart2000] (1981 - 2019; n = `r printnum(length(unique(wvs$S009)))` nations). We downloaded the full Integrated Values Survey, which included all waves from the World Values Survey and the European Values Survey, and computed the two dimensions of cultural values following procedures from previous research[@Inglehart2000]. Additionally, we retrieved cross-sectional data on cultural tightness (n = `r printnum(nrow(tightness))` nations) and individualism (n = `r printnum(nrow(fincherData))` nations) from previous work[@Gelfand2021;@Fincher2008].

To calculate geographic and cultural phylogenetic signal, we created two proximity matrices for 269 of the world's nations: a geographic proximity matrix and a linguistic proximity matrix. Geographic distance between two nations was calculated as the logged geodesic distance between nation capital cities (data from the maps R package[@Brownrigg2018]) using the geosphere R package[@Hijmans2019]. The geographic proximity matrix was computed as one minus the log geographic distance matrix scaled between 0 and 1. Linguistic proximity between two nations was calculated as the cultural proximity between all languages spoken within those nations, weighted by speaker percentages. We acquired cultural proximity data by combining the language family trees provided by Glottolog v3.0[@Glottolog] into one global language tree (undated and unresolved). We calculated cultural proximity $s$ between two languages $j$ and $k$ as the distance (in number of nodes traversed) of their most recent common ancestor $i$ to the root of the tree, through the formula:

\begin{gather}
s_{jk} = \frac{n_{r}-n_{i}}{n_{r}}
\end{gather}

where $n_{r}$ is the maximum path length (in number of nodes traversed) leading to the pan-human root $r$, and $n_{i}$ is the maximum path length leading to node $i$. We then combined these proximities with speaker data from Ethnologue 21[@Ethnologue] and compared every language spoken within those nations by at least 1 permille of the population, weighted by speaker percentages, through the formula:

\begin{gather}
w_{lm} = {\Sigma}{\Sigma}p_{lj}p_{mk}s_{jk}
\end{gather}

where $p_{lj}$ is the percentage of the population in nation $l$ speaking language $j$, $p_{mk}$ is the percentage of the population in nation $m$ speaking language $k$, and $s_{jk}$ is the proximity measure between languages $j$ and $k$[@Eff2008]. This calculation resulted in a linguistic proximity matrix with values between 0 and 1.

We included these matrices in Bayesian multilevel models, allowing nation random intercepts to covary according to both geographic and linguistic proximity simultaneously. These models were fitted with the R package brms[@Burkner2017] and converged normally ($\hat{R}$ < 1.1). The assumptions of these models were met: residuals were approximately normally distributed, though this was not formally tested. Estimates of geographic and cultural phylogenetic signal were computed as the proportion of national-level variance in these models explained by geographic and linguistic proximity matrices.

## Literature review

We exported two searches from Web of Science (https://www.webofknowledge.com/) on 27^th^ September 2021, restricting our searches to articles published between 1900 and 2018. The first search was for the terms "economic development" AND ("cross-national" OR "cross-cultural" OR "cross-country"), which returned 965 articles. The second search was for the terms "values" AND ("cross-national" OR "cross-cultural" OR "cross-country"), which returned 6806 articles. As this was not a formal systematic literature review, we did not follow PRISMA[@Page2021] guidelines for systematic literature reviews.

Once exported, we ordered the articles by descending number of citations per year since initial publication, using citation counts reported by Web of Science. We then coded each article, in order, for inclusion in our review. Articles were only included if: (1) they were judged to be relevant to economic development or cultural values; (2) they were an original empirical research article; and (3) they contained at least one analysis with national-level outcome or predictor variables. We stopped when we had included 50 articles for the economic development review and 50 articles for the cultural values review.

Within each included article, we exhaustively coded every individual cross-national analysis reported in the main text. We coded mainly correlation or regression analyses, and explicitly excluded meta-analyses, factor analyses, measurement invariance analyses, multidimensional scaling analyses, hierarchical clustering analyses, multiverse analyses, and scale development / validation analyses. We also excluded analyses that compared only two, three, four, five, or six nations. For each included analysis, we recorded the year, impact factor of the journal (retrieved from https://jcr.clarivate.com/jcr/home), outcome variable, all predictor variables, test statistic, p-value, number of nations, number of data points, model type, if the data were available, and whether and how the analysis attempted to control for non-independence.

We coded common attempts to control for non-independence between nations. These included: (1) any higher-level control variables for spatial regional groupings (e.g. continent fixed effects); (2) any geographic distance control variables (e.g. distance between capital cities, distance from equator, latitude); (3) any control variables capturing shared cultural history (e.g. former colony, legal origin fixed effects, linguistic history, cultural influence); and (4) any other control variables, tests, or approaches that were deemed as attempts to control for non-independence (e.g. eigenvector filtering[@Tiefelsdorf2007], controls for trade-weightings between nations, cross-sectional dependence tests[@DeHoyos2006], separate analyses for subsets of nations). These were coded by the first author.

Once we had compiled our review database, we calculated the proportion of articles attempting to control for non-independence at least once. We also calculated the proportion of articles employing the different types of control listed above at least once: regional fixed effects, distance, shared cultural history, or other. For these proportions, we calculated two-tailed 95% bootstrap confidence intervals with 1,000 bootstrap iterations. Additionally, we predicted the probability of an article attempting to control for non-independence at least once using Bayesian logistic regression, including in separate models log journal impact factor and year of publication as linear and spline predictors, respectively.

For individual analyses, we dealt with the nested nature of the data (analyses nested within articles) by fitting Bayesian multilevel logistic regression models with review type (economic development vs. cultural values) as the sole fixed effect and random intercepts for articles. We fitted these models separately for overall attempts to control for non-independence and split by method type. We report the adjusted proportions with equal-tailed 95% credible intervals (see Results and Supplementary Figure \@ref(fig:plotReview2)). Additionally, we predicted the probability of an analysis attempting to control for non-independence using Bayesian multilevel logistic regression with random intercepts for articles. In separate models, we included log journal impact factor and year of publication as linear and spline predictors, respectively. All Bayesian models were fitted with the brms R package[@Burkner2017]. Our priors were informed by prior predictive checks, and all models converged normally ($\hat{R}$ < 1.1). The assumptions of these models (i.e., binary nested data) were met.

## Simulations

```{r, echo=FALSE}
tar_load(geneticDistances)
```

We simulated data for 236 nations $i$ with varying degrees of spatial or cultural phylogenetic signal for outcome $y$ and predictor $x$ using the following generative model:

\begin{gather}
\begin{bmatrix}y_i\\x_i \end{bmatrix} \sim \text{MVNormal}
\begin{pmatrix}\begin{bmatrix}\alpha_y\\\alpha_x \end{bmatrix},\textbf{S}\end{pmatrix}\\
\alpha_y \sim \text{Normal}(0, \sqrt{\lambda} \cdot \Sigma) \nonumber \\
\alpha_x \sim \text{Normal}(0, \sqrt{\rho} \cdot \Sigma) \nonumber \\
\textbf{S} = 
\begin{pmatrix}\sqrt{1 - \lambda} & 0 \\ 0 & \sqrt{1 - \rho} \end{pmatrix}
\begin{pmatrix}1 & r \\ r & 1\end{pmatrix}
\begin{pmatrix}\sqrt{1 - \lambda} & 0 \\ 0 & \sqrt{1 - \rho} \end{pmatrix} \nonumber
\end{gather}

where $\Sigma$ is a correlation matrix proportional to either geographic or linguistic proximities between nations, $\lambda$ and $\rho$ are autocorrelation parameters that represent the expected spatial or cultural phylogenetic signal for outcome and predictor variables, respectively, and $r$ is the true cross-national correlation between the variables after accounting for autocorrelation. Importantly, when $r = 0$ in this simulation, we know that there is no direct causal relationship between $y$ and $x$. Instead, any relationship between the two variables is merely the result of autocorrelation.

We set the autocorrelation parameters $\lambda$ and $\rho$ to either 0.2 (weak), 0.5 (moderate), or 0.8 (strong). We also initially set the true cross-national correlation to 0 in order to determine false positive rates, and then additionally set $r$ to 0.1 (small effect), 0.3 (medium effect), and 0.5 (large effect) in order to determine statistical power to detect true effects. For each parameter combination, we simulated 100 datasets, resulting in 3600 datasets. Each dataset had 236 rows representing different nations, with the following associated data for each nation: latitude, longitude, continent (Africa, Asia, Europe, North America, Oceania, or South America), language family of the nation's majority spoken language (Afro-Asiatic, Atlantic-Congo, Austroasiatic, Austronesian, Eskimo-Aleut, Indo-European, Japonic, Kartvelian, Koreanic, Mande, Mongolic-Khitan, Nilotic, Nuclear Trans New Guinea, Sino-Tibetan, Tai-Kadai, Tupian, Turkic, or Uralic), the mean of the predictor variable within a 2000km radius, and coordinates for genetic distances from a previous study[@Schulz2019] (only available for `r printnum(geneticDistances %>% drop_na() %>% nrow())` nations).

With the resulting simulated datasets, we standardised outcome and predictor variables and fitted eleven different models: (1) naive regression without controls, (2) regression with latitude control, (3) regression with longitude control, (4) regression with continent fixed effects, (5) regression with language family fixed effects, (6) regression controlling for the mean of the predictor variable in a 2000km radius, (7) regression employing Conley standard errors based on geographic distances, (8) regression employing Conley standard errors based on genetic distances, (9) Bayesian regression including a Gaussian process over latitudes and longitudes, (10) Bayesian regression including random intercepts covarying according to linguistic proximity, and (11) Bayesian regression including both a Gaussian process over latitudes and longitudes and random intercepts covarying according to linguistic proximity.

Models employing Conley standard errors either required latitude and longitude values or coordinates for genetic distances. To determine distance cutoffs, we employed an approach recommended in previous work[@Colella2019]: we fitted models with a range of feasible distance cutoffs and retained the model with the largest standard error for the slope parameter. These models were fitted using the conleyreg R package[@Duben2021]. Bayesian models were fitted using the brms R package[@Burkner2017]. Our choice of priors was based on prior predictive simulation. All models converged normally ($\hat{R}$ < 1.1). Across all model types and parameter combinations, we calculated the false positive rate as the proportion of models that estimated slopes with a two-tailed 95% confidence / credible interval excluding zero when $r = 0$. We calculated statistical power as the proportion of models that estimated slopes with a 95% confidence / credible interval excluding zero when $r > 0$. We calculated two-tailed 95% bootstrap confidence intervals for these false positive rates and statistical power estimates with 1,000 bootstrap iterations.

## Reanalyses

We searched the individual analyses from our literature review for statistically significant cross-national correlations with available primary or secondary data. We restricted our search to one analysis per paper, and searched until we had a set of twelve analyses, six from economic development papers and six from cultural values papers, for which we were able to replicate the original result (i.e. find a cross-national correlation with the same sign and roughly the same effect size). We also ensured that at least one analysis was a multilevel model, with multiple observations per nation.

The twelve analyses that we settled on[@Adamczyk2009;@Alesina2013;@Beck2003;@Beck2005;@Bockstette2002;@Easterly2003;@Easterly2007;@Fincher2008;@Gelfand2011;@Inglehart2000;@Knack1997;@Skidmore2002] were mostly bivariate cross-national correlations, except for two. One analysis[@Gelfand2011] additionally controlled for log gross national income, and another analysis[@Skidmore2002] is a multilevel model including random intercepts for nations and several individual-level and national-level covariates (see Model 5 in original paper). Before running any additional models, we pre-registered these twelve analyses on the Open Science Framework on 25^th^ January 2022 (https://osf.io/u8tbf). We endeavoured to keep the sample sizes of our reanalyses as close to the original analyses as possible, though there were some deviations (see Supplementary Table \@ref(tab:tableDeviations)). Despite these slight deviations from the original analyses, all models reported in Figure \@ref(fig:plotReplications1) are fitted to the same number of data points, meaning that any changes in effect sizes are solely due to controlling for non-independence.

For each individual analysis, we ran four models: (1) a naive regression replicating the original finding, (2) a regression including a Gaussian process allowing nation random intercepts to covary according to a geographic proximity matrix from latitude and longitude values, (3) a regression including nation random intercepts that covaried according to a linguistic proximity matrix, and (4) a regression including both a geographic Gaussian process and nation random intercepts with linguistic covariance. See Supplementary Methods for full models.

We fitted these models using the brms R package[@Burkner2017]. Our choice of priors was based on prior predictive simulation. All models converged normally ($\hat{R}$ < 1.1), though for some models we resorted to using approximate Gaussian processes[@Solin2020] to reach convergence. The assumptions of these models were met: residuals were approximately normally distributed, though this was not formally tested.

## Reproducibility

All data and code are accessible on GitHub[@github]. We used the targets R package[@Landau2021] to create a reproducible data analysis pipeline and the papaja R package[@Aust2020] to reproducibly generate the manuscript.

## Ethics

We did not apply for ethical approval through an ethics board as we analysed only publicly available and simulated data in this study.

\newpage
\nolinenumbers

# Data Availability

All data to reproduce the statistical analyses in this manuscript can be found on GitHub[@github]. Data on human development were retrieved from the United Nations Development Programme (https://hdr.undp.org/en/content/download-data). Data on GDP per capita, annual GDP per capita growth, and the Gini coefficient were retrieved from the World Bank (https://data.worldbank.org/). Data on traditional vs. secular values and survival vs. self-expression values were retrieved from the World Values Survey (https://www.worldvaluessurvey.org/wvs.jsp). Data on cultural tightness were retrieved from the OSF repository for ref[@Gelfand2021] (https://osf.io/47pe8/). The review data generated in this study are provided in Supplementary Data 1. All other datasets (e.g., for replications) were retrieved from tables and supplementary tables directly from papers cited in the main text[@Adamczyk2009;@Alesina2013;@Beck2003;@Beck2005;@Bockstette2002;@Easterly2003;@Easterly2007;@Fincher2008;@Gelfand2011;@Inglehart2000;@Knack1997;@Skidmore2002] - these datasets can be found in our GitHub repository[@github].

# Code Availability

All code to reproduce the statistical analyses in this manuscript can be found on GitHub[@github].

\newpage

# References

\begingroup
<div id="refs" custom-style="Bibliography"></div>
\endgroup

\newpage

# Acknowledgements

This work was supported by a Royal Society of New Zealand Marsden grant (20-UOA123) to QDA.

# Author Contributions Statement

SC and QDA conceived of and designed the study. SC curated the data, produced all code for analysis and visualisation, and wrote the original draft of the manuscript. TK and QDA developed and compiled the geographic and linguistic distance matrices. QDA provided funding and input on manuscript preparation and revision. All authors reviewed and edited the final draft of the manuscript.

# Competing Interests Statement

The authors declare no competing interests.

\newpage
\vspace*{60mm}

```{r child = "appendix.Rmd"}
```