Title page
Contents
EXECUTIVE SUMMARY 4
The Professional Development Program to Support Data-Driven Instruction Intervention 5
Key Findings 5
Concluding Thoughts 8
I. INTRODUCTION 9
Previous Research on DDI Interventions 9
Overview of the Support for DDI Intervention Examined in This Study 12
Focused Schools Consultants' Services 15
Data Coaches and Their Roles in Treatment Schools 19
Research Questions 20
II. STUDY SAMPLE, DESIGN, DATA, AND METHODS 22
Study Sample 22
District and School Samples 22
Principal, Teacher, and Student Samples 24
Data Coach Sample 24
Experimental Design to Estimate the Impact of Support for DDI 24
Approach to Random Assignment 25
Baseline Characteristics of Treatment and Control Schools 25
Data Sources 26
Analytic Approach 28
III. IMPLEMENTATION OF THE PROFESSIONAL DEVELOPMENT PROGRAM TO SUPPORT DATA-DRIVEN INSTRUCTION INTERVENTION 30
The Intervention as Implemented 30
Implementation of Intervention Supports 30
Implementation of Provider Services 31
Implementation of School-Level Activities 32
Contrast Between Treatment and Control Schools in Data-Related Supports and Activities 38
Data-Related Professional Development 38
Guidance and Support From Coaches and School Leaders in Data Use Activities 41
Teacher Collaboration on Data-Related Activities 45
Summary of Implementation Findings 47
IV. IMPACTS OF PROFESSIONAL DEVELOPMENT IN SUPPORT OF DATA-DRIVEN INSTRUCTION 48
Impacts of Support for DDI on Teachers' Practices 48
Teachers' Data Use 48
Teachers' Instructional Strategies 50
Impacts of Support for DDI on Student Achievement 51
Average Impacts on Student Achievement 52
Variation in the Impacts of Support for DDI across Schools 55
Concluding Thoughts 61
REFERENCES 63
ACKNOWLEDGEMENTS 66
DISCLOSURE OF POTENTIAL CONFLICTS OF INTEREST 66
APPENDIX A. SUPPLEMENTAL INFORMATION ON STUDY DESIGN, DATA, AND METHODS 71
Study Sample 72
Study Districts and Schools 72
Samples of Principals, Teachers, and Students 74
Data Coach Sample 80
Matched-Pair School-Level Random Assignment Design 81
Statistical Power 81
Matching of Schools 82
Random Assignment 83
Student Attrition 84
Data Collection 88
Program Monitoring Data 88
Principal and Teacher Surveys 89
Student Administrative Data 89
Analysis Methods 90
Standardizing Test Score Outcomes 90
Estimating Impacts on Student Outcomes 90
Estimating Impacts on Teacher and Principal Outcomes 94
Analysis of Variation in Impacts 95
APPENDIX B. SUPPLEMENTAL FINDINGS ON IMPLEMENTATION OF THE DATA-DRIVEN INSTRUCTION INTERVENTION (CHAPTER III) 99
Implementation of Support for DDI in Treatment Schools 100
Key Intervention Supports and Provider Services 100
Intervention Activities in Treatment Schools 102
Differences between Treatment and Control Schools in Data-Related Activities 105
APPENDIX C. SUPPLEMENTAL FINDINGS ON THE IMPACTS OF THE DATA-DRIVEN INSTRUCTION INTERVENTION (CHAPTER IV) 112
Impacts of the Support for DDI intervention on Intermediate Outcomes 113
Teachers' Access to and Use of Data 113
Principals' Access to and Use of Data 116
Teachers' Use of Instructional Strategies Associated with DDI 118
Impacts of the Support for DDI Intervention on Student Outcomes 119
Estimated Impacts on Student Achievement in Spring 2017 119
Sensitivity of Estimated Impacts on Student Achievement to Alternative Specifications 121
Impacts on Student Behavior 123
Impacts Among Schools with an Instructional Focus that Included English/Language Arts 124
Impacts on Student Achievement, by Student Subgroup 124
Variation Across Districts and School Readiness in Impacts on Student Achievement 126
Variation in Impacts on Student Achievement and Teacher Practices Across Levels of Coach Preparation 128
ACKNOWLEDGEMENTS 131
DISCLOSURE OF POTENTIAL CONFLICTS OF INTEREST 131
Table I.1. Ongoing professional development schedule and topics 18
Table II.1. Data sources for the DDI study 27
Table III.1. Principal professional development during 2015-2016 39
Table III.2. Percentage of teachers who reported guidance or feedback from school leaders on data-related topics at least monthly or several times per term during 2015-2016 43
Figure I.1. Logic model for the support for DDI intervention 14
Figure II.1. States represented by districts in study sample 23
Figure II.2. Baseline characteristics of treatment and control students and teachers 26
Figure III.1. Percentage of principals in treatment schools providing guidance to teachers on aspects of data-driven instruction 33
Figure III.2. Percentage of schools in which instructional leadership teams had completed key components of the data-driven instruction intervention, fall 2015 and spring 2016 35
Figure III.3. Percentage of schools that selected an instructional focus in different areas, 2015-16 36
Figure III.4. Percentage of schools in which data coach engaged in different activities, fall 2015 and spring 2016 37
Figure III.5. Teacher professional development (PD) during 2015-2016 40
Figure III.6. Percentage of teachers who reported working one-on-one with a coach or school leader at least monthly on different aspects of data use during 2015-2016 42
Figure III.7. Percentage of teachers who developed and updated analyses of student progress at least monthly or several times per term during 2015-2016 44
Figure III.8. Percentage of teachers who reported receiving training or coaching on data-related activities in collaboration with other teachers during common planning... 45
Figure III.9. Percentage of teachers who reported engaging in data-related activities with other teachers during common planning periods at least monthly during 2015-2016 46
Figure IV.1. Percentage of grade 4 and 5 teachers who reported using data daily or several times per week for different purposes in math during 2015-2016 50
Figure IV.2. Percentage of grade 4 and 5 teachers who reported using data daily or several times per week for different purposes in English/language arts during 2015-2016 50
Figure IV.3. Percentage of grade 4 and 5 teachers who reported using instructional practice daily or several times per week for different purposes in English/language arts and... 51
Figure IV.4. Mean student achievement on 2016 state assessments in math and English/language arts 52
Figure IV.5. Mean student achievement on 2016 state assessments in math and English/language arts, among schools with an instructional focus in an area of English/language arts 54
Figure IV.6. Estimated impact of Support for DDI on student achievement on 2016 state assessments in math, by matched school pair 56
Figure IV.7. Estimated impact of Support for DDI on student achievement on 2016 state assessments in English/language arts, by matched school pair 57
Figure IV.8. Estimated impact of support for DDI on student achievement on 2016 state assessments in math and English/ language arts, by school's implementation readiness 60
Boxes
Box 1. What were schools trained to do with student data? 16
Box 2. What information was given to schools on evidence-based strategies to improve instruction? 17
Box 3. What kinds of instructional focus and best practices did instructional leadership teams choose? 36
Box 4. Were the impacts of support for DDI in a school related to implementation readiness? 58
Table A.1. Comparison of study districts to all districts and largest districts in the United States 74
Table A.2. Minimum detectible effects in study design 81
Table A.3. Characteristics of treatment and control schools 83
Table A.4. Attrition from the spring 2016 and spring 2014 enrollment samples 86
Table A.5. Baseline characteristics of students in the analysis sample (those enrolled in a study school in spring 2016 and with valid test scores) 87
Table A.6. Student mobility into analysis sample, by treatment group 88
Table A.7. Rates of missing data on model covariates in the main impact model and sensitivity model with additional covariates 92
Table B.1. Data coaches' background 100
Table B.2. Attendance at and ratings of professional development (PD) sessions 101
Table B.3. Data coach assessment of usefulness of data-driven instruction training 102
Table B.4. Data coach assessment of principal's level of support of data-driven instruction 103
Table B.5. Instructional focus and best practices chosen by instructional leadership teams 103
Table B.6. Percentage of coaches that reported challenges with teams during 2015-2016 104
Table B.7. Teacher professional development during 2015-2016, by teacher type 105
Table B.8. Instructional leadership team activities as reported by principals during 2015-2016 107
Table B.9. Percentage of teachers who reported guidance or feedback from school leaders on data-related topics at least weekly or several times per month during 2015-2016 109
Table C.1. Impacts of support for DDI on teachers' reports of access to specific types of student data 114
Table C.2. Impacts of support for DDI on the types of data teachers used to guide math and reading instruction (percentages who reported using each type of data at least... 115
Table C.3. Impacts of support for DDI on teachers' reports that specific types of student data are "very useful" when making instructional decisions 116
Table C.4. Impacts of support for DDI on teachers' level of confidence in using data to guide their instruction 116
Table C.5. Impacts of support for DDI on the types of student data principals used to inform school decisions (percentages who reported at least monthly or several times per term) 117
Table C.6. Impacts of support for DDI on principals' reported barriers to accessing student data (percentages who reported a moderate or major barrier) 118
Table C.7. Impacts of support for DDI on 2016 and 2017 student achievement 121
Table C.8. Impacts of support for DDI on students' math scores 122
Table C.9. Impacts of Support for DDI on students' English/language arts scores 123
Table C.10. Impacts of support for DDI on student behavioral outcomes 124
Table C.11. Impact of support for DDI on student achievement and teacher practices in math and English/language arts, by coach's prior experience as instructional coach 129
Table C.12. Impact of support for DDI on student achievement and teacher practices in math and English/language arts, by coach-reported adequacy of DDI training 130
Figure A.1. Description of principal analysis sample 75
Figure A.2. Description of teacher analysis sample 76
Figure A.3. Description of student analysis sample: Math achievement 2014 77
Figure A.4. Description of student analysis sample: ELA achievement 2014 78
Figure A.5. Description of student analysis sample: Math achievement 2016 79
Figure A.6. Description of student analysis sample: ELA Achievement 2016 80
Figure B.1. Turnover among key staff in treatment schools 101
Figure B.2. Frequency of meetings between principal and data coach, 2015-2016 102
Figure B.3. Frequency of teacher collaboration team meetings, 2015-2016 104
Figure B.4. Percentage of principals providing guidance to teachers on aspects of data-driven instruction at least monthly or several times per term during 2015-2016 108
Figure B.5. Percentage of teachers who reported working one on one with a coach or school leader on different aspects of data use at least weekly or several times... 108
Figure B.6. Planning by School Leaders of Data-Related Training or Structured Supports for Teachers during 2015-2016 109
Figure B.7. Percentage of teachers who reported receiving training or coaching on data-related activities in collaboration with other teachers during common planning... 110
Figure B.8. Percentage of teachers who reported collaboration with other teachers during common planning periods on data-related activities at least weekly or several times... 111
Figure C.1. Percentage of grade 4 and 5 teachers who reported using data at least several times per month for different purposes in math during 2015-2016 113
Figure C.2. Percentage of grade 4 and 5 teachers who reported using data at least several times per month for different purposes in English/language arts during 2015-2016 114
Figure C.3. Percentage of principals who used data at least monthly or several times per term for different purposes during 2015-2016 117
Figure C.4. Percentage of grade 4 and 5 teachers who reported using instructional practice at least several times per month for different purposes in English/language arts... 119
Figure C.5. Mean student achievement on 2016 state assessments in math and English/language arts, among schools with an instructional focus in an area of English/language... 124
Figure C.6. Impacts of support for DDI on students' math and English/language arts scores, by grade level 125
Figure C.7. Impacts of support for DDI on students' math and English/language arts scores, by students' baseline proficiency level 125
Figure C.8. Estimated impact of support for DDI on student achievement on 2016 state assessments in math, by district 126
Figure C.9. Estimated impact of support for DDI on student achievement on 2016 state assessments in English/language arts, by district 127
Figure C.10. Estimated impact of support for DDI on student achievement on 2016 state assessments in math and English/ language arts, by school's implementation... 128