Estimation of country-level effects in cross-national survey research using multilevel modelling: The role of statistical power (Preprint)

Damian, E., Meuleman, B., & van Oorschot, W. (2022, August 19). Estimation of country-level effects in cross-national survey research using multilevel modelling: The role of statistical power. https://doi.org/10.31219/osf.io/m94kh

Multilevel regression analysis is one of the most popular types of analyses in cross-national social studies. However, since its early applications, there have been constant concerns about the relatively small numbers of countries in cross-national surveys and its ability to produce unbiased and accurate country-level effects. A recent review of Bryan and Jenkins (2016) highlights that there are still no clear rules of thumb regarding the minimum number of countries needed. The current recommendations vary from 15 to 50 countries, depending on model complexity. This paper aims to offer a better understanding regarding the consequences of group-level sample size, model complexity, effect size, and estimator procedure on the precision to estimate country-level effects in cross-national studies. The accuracy criteria considered are statistical power, relative parameter bias, relative standard error bias, and convergence rates. We pay special attention to statistical power - a key criteria that has been largely neglected in past research. The results of our Monte Carlo simulation study indicate that the small number of countries found in cross-national surveys seriously affects the accuracy of group-level estimates. Specifically, while a sample size of 30 countries is sufficient to detect large population effects (.5), the probability of detecting a medium (.25) or a small effect (.10) is .4 or .2, respectively. The number of additional group-level variables (i.e., model complexity) included in the model does not disturb the relationship between sample size and statistical power. Hence, adding contextual variables one by one does not increase the power to estimate a certain effect if the sample size is small. Even though we find that Bayesian models have more accurate estimates, there are no notable differences in statistical power between Maximum Likelihood and Bayesian models.