Title page
Contents
ACKNOWLEDGMENTS 4
INTRODUCTION 9
1. SMALL AREA ESTIMATION PROCESS OVERVIEW 11
2. SURVEY ESTIMATES 14
2.1. Raking 14
2.2. Generate Survey Estimates 16
2.3. Variance Smoothing for Averages 18
3. MODEL ESTIMATION AND PREDICTION 22
3.1. Covariate Formation 22
3.1.1. Variable Modification 22
3.1.2. Final Reduction of Covariates 23
3.2. Modeling 26
3.2.1. Modeling Proportions - Area-Level Univariate HB Linear Model 26
3.2.1.1. Arcsine Square Root Transformation 27
3.2.1.2. Model Specification 27
3.2.2. Modeling Averages - Area-Level Univariate HB Linear Model 28
3.2.3. Model Estimation 29
3.2.4. Predicted Values 29
3.2.5. Measures of Precision for the Model-Based Estimates 30
4. BENCHMARKING 32
5. MODEL DIAGNOSTICS AND EVALUATION 33
5.1. Internal Model Validation 33
5.1.1. Convergence and Mixing Diagnostics 33
5.1.2. Multicollinearity Test 34
5.1.3. Residual Analysis 35
5.1.4. Posterior Predictive Checks 36
5.2. External Model Validation 37
5.2.1. Histograms of Differences in Estimates 38
5.2.2. Shrinkage Plots 38
5.2.3. Interval Coverage Plots 39
5.2.4. Bubble Plots of Survey Estimates and Model-Based Estimates 40
5.2.5. Survey Estimates and Model-Based Estimates Variances 41
6. SUMMARY 43
REFERENCES 45
Table 1-1. State-level sample size distributions for age and education groups 12
Table 2-1. Median and interquartile range (IQR) of raking adjustment factors, by state, sorted by descending median 15
Table 2-2. Distribution of the proportion of variance associated with multiple imputation for survey estimates across groups, across states: 2012/2014/2017 PIAAC 18
Table 2-3. Summary of average proficiency score variance estimates prior to and after smoothing: 2012/2014/2017 PIAAC 20
Table 3-1. List of covariates for the small area models, initially considered for the state model and finally selected 24
Table 3-2. Total of Akaike information criterion (AIC) measures over six age groups for fitting different outcomes and the total across all six outcomes 25
Table 3-3. Total of Akaike information criterion (AIC) measures over four education groups for fitting different outcomes and the total across all six outcomes 26
Table 3-4. Regression coefficients and components of the variance-covariance matrices of random effects for the final Hierarchical Bayes (HB) model: for the... 29
Table 3-5. Distribution of credible interval widths and coefficients of variation for state-level model-based estimates for the less than high school group and... 31
Table 5-1. Convergence diagnostics for the Markov chain Monte Carlo (MCMC): 2012/2014/2017 PIAAC 34
Table 5-2. Variance inflation factors (VIFs) 34
Table 5-3. Summaries of posterior predictive statistics for literacy proportions at or below Level 1: 2012/2014/2017 PIAAC 37
Figure 1-1. Overview of the modeling process 13
Figure 5-1. Residual plots for the first set of residuals 35
Figure 5-2. Residual plots for the second set of residuals (conditional on the random effect) 36
Figure 5-3. Literacy proportions (less than high school) - Histograms of differences between survey estimates and model-based estimates: 2012/2014/2017 PIAAC 38
Figure 5-4. Literacy proportion (less than high school) - Shrinkage plots of point estimates by sample size: 2012/2014/2017 PIAAC 39
Figure 5-5. Literacy proportion (less than high school) - Indication of coverage by credible interval: 2012/2014/2017 PIAAC 40
Figure 5-6. Literacy proportion (less than high school) - Comparison between survey estimates and model-based estimates: 2012/2014/2017 PIAAC 41
Figure 5-7. Literacy proportion (less than high school) - Comparison between model standard errors and survey standard errors: 2012/2014/2017 PIAAC 42