Title page
Contents
Acknowledgments 4
Disclosure of Potential Conflicts of Interest 5
Executive Summary 11
1. Introduction 16
2. Evaluation of the OSP 18
The Sample: Number of Applicants and Scholarship Awards by Lotteries 19
Data Sources 20
Approach for Measuring Impacts 22
Limitations 23
3. Characteristics of the Program, Students, and Schools 24
Program Features 24
Characteristics of Students Applying to the OSP 25
Student Participation in the Program 27
School Characteristics 28
4. Impacts on Key Outcomes 34
Impacts on Reading and Mathematics Achievement 34
Impacts on Parent and Student Satisfaction 41
Impacts on Parent and Student Perceptions of School Safety 43
Impacts on Parent Involvement in Education 45
References 47
Appendix A. Lottery Structure, Study Sample, and Impact Findings 49
A-1. Lottery Structure 49
A-2. Characteristics of the Study Sample 50
A-3. Impact Findings by Outcome and Student Subgroups 55
Appendix B. Technical Approach 63
B-1. Measuring the Impact of a Scholarship Offer and Its Use 63
B-2. Detecting Impacts 63
B-3. Estimating Impacts 66
B-4. Method for Calculating Percentile Changes 68
B-5. Outcome Measures and Data Collection Procedures 70
B-6. Sampling and Nonresponse Weights 72
Appendix C. Additional Analyses 77
C-1. Impacts on Test Scores in SINI and Non-SINI Schools, Excluding Pre-K Students 77
C-2. Alternative Definitions of Scholarship Use 78
C-3. Sensitivity Analysis for School Safety as Reported by Students 79
C-4. Comparing Impacts Between the Study's Two Followup Years 81
C-5. Mediation Analysis of School Choice and Parent Satisfaction 82
C-6. Supplemental Tables 84
Table 1. OSP scholarship offers by study cohort 19
Table 2. Data sources used to estimate impacts 20
Table 3. Study cohorts and years tested 21
Table 4. Percentage of study participants in the second-year impact sample, by school type, two years after applying to the program 30
Table 5. School attendance patterns during the first two years for students in the second-year impact sample 31
Table 6. Characteristics of schools that students in the second-year impact sample attended, two years after application 33
Table 7. Percentage of parents giving their child's school a grade of A or B, by whether they exercised choice 43
Figure E-1. Impacts on reading and mathematics achievement (percentile scores) for scholarship offer and use, in second year 12
Figure E-2. Impacts on parent and student general satisfaction (percentage giving school an A or B grade) for scholarship offer and use, in second year 13
Figure E-3. Impacts on parent and student general perceptions of school safety (percentage rating school as very safe) for scholarship offer and use, in second year 14
Figure E-4. Impacts on parent involvement in education at school and at home (number of events reported) for scholarship offer and use, in second year 15
Figure 1. Percentage of eligible program applicants, by SINI status and school grade level at time of application 25
Figure 2. Percentage of eligible applicants, by entering grade level at time of application 26
Figure 3. Percentage of eligible applicants, by school type at time of application 26
Figure 4. Percentage of treatment group students in the second-year impact sample using the scholarship after application, by number of semesters 27
Figure 5. Percentage of treatment group students in the second-year impact sample using the scholarship after application, by semester 28
Figure 6. Percentage of participating private schools, by religious affiliation and tuition rates 29
Figure 7. Percentage of participating private schools, by the share of OSP students enrolled in their school 29
Figure 8. Impacts on reading and mathematics achievement (percentile scores) for scholarship offer and use, in second year 34
Figure 9. Impacts on reading achievement (percentile scores) for scholarship offer and use, for students previously attending SINI and non-SINI schools, in second year 36
Figure 10. Impacts on mathematics achievement (percentile scores) for scholarship offer and use, for students previously attending SINI and non-SINI schools, in second year 37
Figure 11. Impacts on reading achievement (percentile scores) for scholarship offer and use, for students at elementary and secondary schools, in second year 38
Figure 12. Impacts on mathematics achievement (percentile scores) for scholarship offer and use, for students at elementary and secondary schools, in second year 38
Figure 13. Impacts on reading achievement (percentile scores) for scholarship offer and use, for students below and above median for reading achievement at time of application, in second year 39
Figure 14. Impacts on reading achievement (percentile scores) for scholarship offer and use, for students below and above median for mathematics achievement at time of application, in second year 40
Figure 15. Impacts on mathematics achievement (percentile scores) for scholarship offer and use, for students below and above median for reading achievement at time of application, in second year 40
Figure 16. Impacts on mathematics achievement (percentile scores) for scholarship offer and use, for students below and above median for mathematics achievement at time of application, in second year 41
Figure 17. Impacts on parent and student general satisfaction (percentage giving school an A or B grade) for scholarship offer and use, in second year 42
Figure 18. Impacts on parent and student general perceptions of school safety (percentage rating school as very safe) for scholarship offer and use, in second year 44
Figure 19. Impacts on parent involvement in education at school and at home (number of events reported) for scholarship offer and use, in second year 46
Exhibits
Exhibit 1. Overview of the Opportunity Scholarship Program as defined in the SOAR Act 16
Exhibit 2. Evaluation questions 18
Table A-1. Scholarship offers by priority group categories, by application year and treatment status 49
Table A-2. Characteristics of treatment and control groups at time of application (full sample) 50
Table A-3. Sample size, valid sample, and percentage missing data at second-year followup 51
Table A-4. Characteristics of treatment and control groups at time of application, for students who completed reading tests at second-year followup 52
Table A-5. Characteristics of treatment and control groups at time of application, for parents who completed surveys at second-year followup 53
Table A-6. Characteristics of treatment and control groups at time of application, for students who completed surveys at second-year followup 54
Table A-7. Impact estimates of the offer and use of a scholarship on reading test scores after two years 55
Table A-8. Impact estimates of the offer and use of a scholarship on mathematics test scores after two years 56
Table A-9. Impact estimates of the offer and use of a scholarship on parent general satisfaction after two years 57
Table A-10. Impact estimates of the offer and use of a scholarship on student general satisfaction after two years 58
Table A-11. Impact estimates of the offer and use of a scholarship on parent general perceptions that school is very safe after two years 59
Table A-12. Impact estimates of the offer and use of a scholarship on student general perceptions that school is very safe after two years 60
Table A-13. Impact estimates of the offer and use of a scholarship on parent involvement in school after two years 61
Table A-14. Impact estimates of the offer and use of a scholarship on parent involvement at home after two years 62
Table B-1. Minimum detectable effect sizes 65
Table B-2. Comparison of primary regression and polynomial model estimates of the impacts of offering a scholarship on reading and mathematics achievement in Year 2 67
Table B-3. Comparison of Bloom adjustment and instrumental variables estimates of the impacts of using a scholarship (TOT estimates) on reading and mathematics achievement in Year 1 68
Table B-4. Computing percentile changes, by grade level, reading 69
Table B-5. Student test response rates for second-year followup 71
Table B-6. Parent survey response rates for second-year followup 71
Table B-7. Student survey response rates for second-year followup 71
Table B-8. Student reading tests 74
Table B-9. Student mathematics tests 74
Table B-10. Parent survey 75
Table B-11. Student survey 75
Table C-1. Comparing subgroup impacts with and without pre-K students in the sample 78
Table C-2. Comparison of treatment impacts using two approaches for TOT 79
Table C-3. Significant coefficients from model of response to student survey 80
Table C-4. Sensitivity of student safety impact estimate to dropping covariates 81
Table C-5. Comparing test score impacts in the first and second years (students tested in both years only) 82
Table C-6. Results of mediation analysis of effects of choice on parent satisfaction 83
Table C-7. Percentage of parents reporting satisfaction with specific aspects of their child's school 84
Table C-8. Percentage of students reporting negative safety incidents that occurred at school 86
Table C-9. Percentage of parents reporting involvement in education activities at school 88
Table C-10. Percentage of parents reporting involvement in education activities at home 89