Title page
Contents
Introduction 6
Choose your own adventure 8
Using probability to measure uncertainty 9
Three probability distributions that can help us interpret research evidence 9
BASIE Step 1: Select prior evidence 11
BASIE Step 2: Report impact estimates 14
BASIE Step 3: Interpret impact estimates 14
Cutoffs and characterizations 16
Credible intervals 17
BASIE Step 4: Sensitivity analysis 18
The BASIE probability tool 21
Example setup 21
Specify prior distribution in the BASIE probability tool 21
Input traditional impact estimate 22
Specify simulation precision 23
Shrunken estimate, credible intervals, and posterior probabilities 23
Findings and sensitivity analyses 24
Incorporating sensitivity analysis when designing a study 25
Future directions 26
Local Stop: From prior to posterior probabilities: a closer look 27
Prior distribution 27
Probability distribution of the impact estimate 28
Posterior distribution 30
Local Stop: Why we do not recommend the flat prior 34
Local Stop: Bayesian meta-regression of prior evidence 35
Conceptual approach 35
Technical description of the Bayesian meta-regression model 35
Local Stop: Adjustments for small-study effects 41
Description of our adjustment method 42
Local Stop: Prior distributions ready to use 44
All prior distributions 44
Comparing the normal and skewed generalized t-distributions 49
Local Stop: Misinterpretations to avoid 53
Local Stop: Power analysis 54
Simulation framework for calculating power and the MDE 54
Examples 55
Local Stop: Monte Carlo simulation approach used by the BASIE probability tool 57
References 58
Appendix A. Additional details regarding our method for adjusting prior evidence for small-study effects 61
Estimating the variance of the maximum order statistic 61
Simulation study 62
Appendix B. Uncertainty arising from less rigorous designs 68
Sources of information regarding the potential magnitude of bias 68
Incorporating uncertainty due to bias into posterior probabilities 69
Exhibits
Exhibit 1. Summary of key steps to applying BASIE 7
Exhibit 2. Pathways for readers of this guide 8
Exhibit 3. The overall distribution of WWC intervention effects 13
Exhibit 4. Report enough posterior probabilities to avoid misleading readers 15
Exhibit 5. Characterization of probabilities 17
Exhibit 6. Sensitivity of a posterior probability to different prior distributions 20
Exhibit 7. Estimated impacts of Crank It Out! on math and reading test scores 21
Exhibit 8. Screenshot of prior selection drop-down menus 21
Exhibit 9. Screenshot of prior probabilities 22
Exhibit 10. Screenshot of estimates entered by the user 22
Exhibit 11. Screenshot of simulation control parameters 23
Exhibit 12. Screenshot of shrunken estimate, credible intervals, and posterior probabilities 24
Exhibit 13. Impacts of Crank It Out! on math and reading test scores 25
Exhibit 14. Results of sensitivity analyses 25
Exhibit 15. Using sensitivity analysis at the design stage 26
Exhibit L1. The overall distribution of WWC intervention effects 28
Exhibit L2. Distribution of the impact estimate assuming the true effect is zero 29
Exhibit L3. Illustration of the relationship between the prior distribution of true effects and the distribution of impact estimates 31
Exhibit L4. The R program used to calculate the posterior distribution illustrated in Exhibit L3 32
Exhibit L5. Posterior distribution of true intervention effects given an impact estimate of 0.15, a standard error of 0.10, and the distribution... 33
Exhibit L6. Prior distributions for all model parameters 40
Exhibit L7. WWC outcome domains included in each of our outcome domains 44
Exhibit L8. Prior distributions with adjustment for small-study effects 46
Exhibit L9. Prior distributions without adjustment for small-study effects 47
Exhibit L10. Density plots of all prior distributions 48
Exhibit L11. R code to create zero-centered prior distribution 49
Exhibit L12. Distribution of all intervention effects in the WWC, estimated using either the normal or skewed generalized t-distribution 51
Exhibit L13. Sensitivity of select posterior probabilities to using either the normal or skewed generalized t-distribution 52
Exhibit L14. Examples of how MDEs vary by prior distribution 55
Appendix Exhibits
Exhibit A1. R code to estimate h(si, d) 62
Exhibit A2. Selection mechanisms 64
Exhibit A3. Tabular comparison of procedures to adjust for reporting bias 66
Exhibit A4. Graphical comparison of procedures to adjust for reporting bias 67