Title page
Contents
THE STUDY'S PRINCIPAL PROFESSIONAL DEVELOPMENT PROGRAM DID NOT AFFECT STUDENT ACHIEVEMENT OR MOST SCHOOL OR TEACHER OUTCOMES MEASURED BY THE STUDY 5
THE PROFESSIONAL DEVELOPMENT PROGRAM HAD SOME NEGATIVE EFFECTS ON PRINCIPALS' PRACTICES 8
THE PROGRAM INCREASED THE AMOUNT OF PROFESSIONAL DEVELOPMENT THAT PRINCIPALS RECEIVED 11
REFERENCES 16
ACKNOWLEDGEMENTS 17
APPENDIX A. THE STUDY'S PRINCIPAL PROFESSIONAL DEVELOPMENT PROGRAM 27
A. Focus and structure of the professional development program 28
B. Characteristics of professional development program staff 33
C. Implementation support for the study's principal professional development program 34
D. Costs of the study's principal professional development program 34
APPENDIX B. STUDY DESIGN, DATA COLLECTION, AND ANALYTIC METHODS 35
A. Study design 36
1. Sample selection 36
2. Random assignment 38
B. Data collection 40
C. Analytic methods 42
1. Constructing outcome measures 42
2. Estimating effects 48
3. Estimating the relationship between characteristics of the professional development program, its effects on principals' practices, and its effects on student achievement 51
APPENDIX C. SUPPLEMENTAL TABLES 52
A. Additional details on findings in the report 53
1. Effects on student achievement and school and teacher outcomes 53
2. Effects on principals' practices 57
3. Implementation of the study's principal professional development program and the overall professional development that principals received from the study and other sources 65
4. Effects on student achievement by district and blocks of schools 68
5. Relationship between characteristics of the professional development program and its effects on principal practices and student achievement 73
B. Additional findings that are not in the report 78
1. Effects on principals' practices 78
2. Implementation of the study's principal professional development program and the overall professional development that principals received 81
C. Supplemental information for systematic reviews 86
D. Realized minimum detectable effects 92
APPENDIX D. INTERPRETING STUDY FINDINGS: ALTERNATIVES TO P-VALUES AND STATISTICAL SIGNIFICANCE 94
A. BASIE (BAyeSian Interpretation of Estimates) 95
B. Interpretation of study findings using the BASIE approach 96
1. Selecting and synthesizing prior evidence 96
2. Findings 97
3. Sensitivity analysis 98
ENDNOTES 102
REFERENCES 104
ACKNOWLEDGEMENTS 106
DISCLOSURE OF POTENTIAL CONFLICTS OF INTEREST 107
Table 1. Teachers' perceptions of the quality of principals' instructional support, by year and participation in the professional development program 10
Figure 1. How the professional development for principals was intended to affect student achievement 3
Figure 2. Student achievement, by year and participation in the professional development program 5
Figure 3. Principal and teacher reports on school climate, by year and participation in the professional development program 6
Figure 4. Principal and teacher retention, by year and participation in the professional development program 7
Figure 5. Hours per week principals spent on leadership activities, by content area and participation in the professional development program 8
Figure 6. Frequency of principals' instructional leadership activities, by year and participation in the professional development program 9
Figure 7. Planned and actual amounts of professional development for principals participating in the program 11
Figure 8. Focus of coaching goals for principals participating in the professional development program 12
Figure 9. Overall number of hours of professional development that principals received, by year and assignment to the program 13
Exhibits
Exhibit 1. The study design 4
Table A.1. Components of the professional development program 28
Table A.2. Percentage of time spent on three school leadership areas, by program component and overall 29
Table A.3. Planned content of summer institute, group training sessions, and professional learning community sessions 29
Table A.4. Characteristics of staff delivering the program 34
Table B.1. Comparison of study districts and public school districts nationally 37
Table B.2. Comparison of study schools and public elementary schools nationally 38
Table B.3. Comparison of baseline characteristics of treatment and control students and schools 39
Table B.4. Comparison of baseline characteristics of principals and teachers in treatment and control schools 39
Table B.5. Data sources 40
Table B.6. Response rates for data sources used to estimate effects 41
Table B.7. Overall model fit 43
Table B.8. Items in final scales and estimated reliabilities 43
Table B.9. Covariates included in the models used to estimate effects 49
Table B.10. Analysis samples 49
Table B.11. Subgroups examined 50
Table B.12. Treatment group characteristics examined 51
Table C.1. Effects on student achievement (supplement to report Figure 2) 54
Table C.2. Effects on student achievement, by principal experience level (supplement to report text) 54
Table C.3. Effects on student achievement, by average baseline achievement of study schools (supplement to report text) 55
Table C.4. Effects on school climate (supplement to report Figure 3) 56
Table C.5. Effects on principal and teacher retention (supplement to report Figure 4) 57
Table C.6. Effects on principals' time use (supplement to report Figure 5 and report text) 58
Table C.7. Effects on duration of teacher observations (supplement to report text) 59
Table C.8. Effects on principals' instructional leadership practices (supplement to report Figure 6, Table 1, and report text) 60
Table C.9. Effects on teacher-reported principals' instructional leadership practices, by principal experience level and school average baseline achievement (supplement to report text) 62
Table C.10. Effects on principals' human capital management and organizational leadership practices (supplement to report text) 63
Table C.11. Amount of professional development and other supports that principals received (supplement to report Figure 9) 66
Table C.12. Content of supports that principals received (supplement to report text) 67
Table C.13. Correlations between the professional development program's effects on principals' instructional leadership practices and its effects on student achievement (supplement to report text) 74
Table C.14. Correlations between the professional development program's effects on principals' human capital management practices and its effects on student achievement (supplement to report text) 75
Table C.15. Correlations between the professional development program's effects on principals' organizational leadership practices and its effects on student achievement (supplement to report text) 75
Table C.16. Correlations between characteristics of the professional development program and its effects on principals' leadership practices (supplement to report text) 76
Table C.17. Effects on principals' leadership practices, by characteristics of the professional development program 77
Table C.18. Effects on principals' time use (lower and upper bounds) 79
Table C.19. Effects on teacher-reported principals' human capital management and organizational leadership practices, by principal experience level and school average baseline achievement 80
Table C.20. Principals' perceptions of the summer institute, group training sessions, and professional learning community sessions, Year 1 82
Table C.21. Principals' perceptions of coaching 83
Table C.22. Usefulness of professional development and other supports that principals received 84
Table C.23. Principals' perceptions of supports received 85
Table C.24. Attrition of study schools, by outcome 87
Table C.25. Missing outcome data, by outcome and year 88
Table C.26. Effects of professional development program, by outcome and year 89
Table C.27. Missing student achievement outcome data, alternate student sample 90
Table C.28. Effects of professional development program on student achievement, alternate student sample 91
Table C.29. Realized minimum detectable effects for key outcomes 92
Table D.1. Prior evidence from the What Works Clearinghouse 97
Table D.2. Effects on student achievement and the probability that the professional development program had effects of a certain size 98
Table D.3. Effects on frequency of instructional support and feedback from principal and the probability that the professional development program had effects of a certain size 98
Table D.4. Sensitivity to the prior evidence for effects on student achievement 99
Table D.5. Sensitivity to the prior evidence for the effects on frequency of instructional support and feedback from principal 100
Table D.6. Difference between traditional effect estimates and Bayesian shrunken estimates 101
Figure A.1. Percent of coaching time spent on meetings and practical applications of coaching 32
Figure A.2. The four phases of the inquiry cycle 33
Figure B.1. Results from district recruitment effort 36
Figure C.1. Effects on students' English language arts achievement in Year 1, by district and random assignment block 68
Figure C.2. Effects on students' math achievement in Year 1, by district and random assignment block 69
Figure C.3. Effects on students' English language arts achievement in Year 2, by district and random assignment block 70
Figure C.4. Effects on students' math achievement in Year 2, by district and random assignment block 71
Figure C.5. Effects on students' English language arts achievement in Year 3, by district and random assignment block 72
Figure C.6. Effects on students' math achievement in Year 3, by district and random assignment block 73
Appendix Exhibits
Exhibit 1. Focus of the study's principal professional development program 28