Title page
Contents
Acknowledgements 8
1. The Investing in Innovation (i3) Fund 9
1.1. The i3 Program Logic Model 10
1.1.1. i3 Program Priorities 10
1.1.2. Key Program Activities 11
1.1.3. i3 Program Goals 13
1.2. Roadmap to this Report 13
2. Implementation of the i3 Program Priorities 14
2.1. Fund Tiers of Grants 14
2.2. Require Independent Evaluations 14
2.3. Fund a Broad Portfolio of Interventions 15
2.3.1. Key Components Implemented by the Intervention Developers 16
2.3.2. Mediators through which Interventions are Expected to Improve Short- and Longer-Term Student Outcomes 17
2.3.3. Short- and Longer-Term Student Outcomes 18
3. Implementation of the Three Key Program Activities Related to Evaluation 21
3.1. Adopt or Develop Criteria to Assess the Strength of the i3 Evaluations 21
3.2. Support the Design and Conduct of the i3 Evaluations 23
3.2.1. Conduct an Initial Assessment of the Need for Support 24
3.2.2. Provide Technical Assistance to Strengthen Design and Conduct of i3 Evaluations 24
3.2.3. Identify and Resolve Risks to the Strength of the Evaluations 27
3.2.4. Periodic Assessments of the Strength of the Evaluations and Reports to the Evaluators and ED 27
3.3. Collect Data on the Conduct of and Findings from the i3 Evaluations 28
3.3.1. Prioritizing the Impact Evaluation Findings 29
3.3.2. Identifying a Finding as the Result of a Prespecified Analysis 30
3.3.3. Using a Survey to Collect Information about the Conduct of and Findings from the Impact and Implementation Evaluations 30
4. Did the i3-Funded Evaluations Meet the Goals of i3? 31
4.1. Were the i3 Evaluations Strong? 31
4.1.1. Were the i3 Implementation Evaluations High Quality? 32
4.1.2. Were the i3 Impact Evaluations Independent? 32
4.1.3. Were the i3 Impact Evaluations Strong? 33
4.1.4. Were the i3 Impact Evaluation Samples Adequately Representative of those Served by each i3 Intervention? 36
4.1.5. Summary: Did the i3 Evaluations Meet the Short-term Goal of i3? 39
4.2. What did the i3 Evaluations Find? 40
4.2.1. Did the i3 Implementation Evaluations Find that the Interventions were Implemented with Adequate Fidelity? 40
4.2.2. Did the i3 Impact Evaluations Find that the i3 Interventions Improved Student Academic Outcomes? 42
4.2.3. Summary: Did the i3 Evaluations Meet the Long-term Goal of i3? 46
5. Did the Scale-up Grants Meet their Proposed Scale-up Goals? 47
5.1. Capacity of the Scale-up Grantees Pre-i3 Award 47
5.2. Assessing whether Scale-up Grants met their Self-Established Scale-up Goals 48
5.3. Description of Implementation of the Scale-up Mechanisms 49
5.3.1. Scale-up Mechanisms 49
5.3.2. Use of Scale-up Funds for Financial Incentives to Support Scale-up 50
6. Summary and Lessons Learned 52
6.1. Summary 52
6.2. Lessons Learned for Future Grant Programs 52
6.2.1. Building on Existing Evidence Standards made Expectations for and Assessments of the Impact Evaluations Clear 52
6.2.2. Developing Standards to Assess ED's Additional Expectations Provided More Information about the Strength of the Evaluations 53
6.2.3. Future Grant Programs Could Find it Challenging to Assess the Strength of Evaluation Designs Submitted after Grant Award 54
6.2.4. Expecting Evaluators to Prespecify Analyses Ensured that Findings from those Analyses would be Reported to the Abt Team, Regardless of the Direction and Magnitude 54
6.2.5. Testing the Interventions at Scale was a Challenge for Scale-up Grants 55
6.2.6. Relying on Administrative Data was a Challenge 56
6.2.7. Resources for Future Programs that Require Evaluation 56
Appendix A: Key Components and Measures of Implementation Fidelity for 67 i3 Interventions 57
Appendix B: i3 Review Protocol, Version 1.0 100
Appendix C: Assessments of and Findings from 67 i3 Impact and Implementation Evaluations 109
Appendix D: Findings Reported by 67 i3 Impact Evaluations from Prespecified Analyses of Student Academic Outcomes 116
Appendix E: Findings Reported by 21 i3 Impact Evaluations from Prespecified Analyses on Non-Academic Outcomes 132
Appendix F: Findings Reported by 40 i3 Impact Evaluations from Analyses that Were Not Prespecified 140
Appendix G: Alternative Approaches to Assessing Representativeness of the 67 i3 Impact Evaluations 154
Appendix H: Short-term and Long-term Goals Met by 67 i3 Impact and Implementation Evaluations 155
Appendix I: Alternative Approaches to Summarizing the Findings from the 67 i3 Impact Evaluations 159
Exhibits
Exhibit 1.1. i3 Program Logic Model 12
Exhibit 2.1. i3 Program Absolute Priorities Identified by the 67 Completed Grants 15
Exhibit 2.2. Generic i3 Intervention Logic Model 16
Exhibit 2.3. Key Components Identified in the i3 Intervention Logic Models 17
Exhibit 2.4. Mediators Identified in the i3 Intervention Logic Models 18
Exhibit 2.5. Short-term Student Outcomes Identified in the i3 Intervention Logic Models 19
Exhibit 2.6. Longer-term Student Outcomes Identified in the i3 Intervention Logic Models 19
Exhibit 2.7. Types of Student Achievement Outcomes Identified in the i3 Intervention Logic Models 20
Exhibit 4.1. Independence of the i3 Impact Evaluations 33
Exhibit 4.2. Modal Evidence Rating Received by Findings from i3 Impact Evaluations 35
Exhibit 4.3. Representativeness of the i3 Impact Evaluations 39
Exhibit 4.4. Number and Percentage of Evaluations that Meet the Short-Term Goal of i3 40
Exhibit 4.5. Summary of Whether the i3 Implementation Evaluations Found that the Interventions Were Implemented with Adequate Fidelity 41
Exhibit 4.6. Summary of Whether the i3 Impact Evaluations Found Statistically Significant Positive Impacts on Student Academic Outcomes 44
Exhibit 4.7. Effect Sizes for i3 Interventions, by Outcome Domain 45
Exhibit 4.8. Number and Percentage of Evaluations that Meet the Long-Term Goal of i3 46
Appendix Exhibits
Exhibit B.1. Relevant Student Outcome Domains Defined for i3 105
Exhibit B.2. Other Student Outcome Domains Defined for i3 106
Exhibit G.1. Representativeness Results under Different Exclusion Thresholds for Schools that Received the i3 Intervention 154
Exhibit I.1. Summary of the Impacts on Student Academic Outcomes Using Our Primary Approach and Two Alternative Approaches 160
Exhibit I.2. Summary of Whether the i3 Impact Evaluations Found Statistically Significant Positive Impacts on Student Academic Outcomes - Based on Modal Findings 161
Exhibit I.3. Summary of Whether the i3 Impact Evaluations Found Statistically Significant Positive Impacts on Student Academic Outcomes - Based on Average Effect Size 162
Exhibit I.4. Summary of Whether the i3 Impact Evaluations Found Statistically Significant Positive Impacts on Student Academic Outcomes - Including Prespecified and Non-Prespecified Analyses 164