Title page
Contents
1. Introduction 20
1.1. The First-Grade and Second-Grade Data Collections 21
1.2. Data Collection Instruments and Administration of Assessments 23
1.3. Contents of Report 25
2. Overview of the Content Coverage and Administration of the Direct Cognitive Assessment Instruments 26
2.1. Content Coverage of the Cognitive Assessments 26
2.1.1. Reading Test Specifications 27
2.1.1.1. Basic Reading Skills 28
2.1.1.2. Vocabulary 30
2.1.1.3. Comprehension 30
2.1.1.4. Continuity Between the ECLS-K and the ECLS-K:2011 Reading Frameworks 31
2.1.2. Mathematics Test Specifications 32
2.1.2.1. Number Properties and Operations 33
2.1.2.2. Measurement 33
2.1.2.3. Geometry 34
2.1.2.4. Data Analysis and Probability 34
2.1.2.5. Algebra 34
2.1.3. Science Test Specifications 35
2.1.3.1. Scientific Inquiry 35
2.1.3.2. Physical Science 35
2.1.3.3. Life Science 36
2.1.3.4. Earth and Space Science 36
2.2. Item and Time Allocation Across and Within Subject Areas 37
2.3. Mode of Administration 39
2.4. Inclusion of Children With Diverse Language Backgrounds and Language of Administration 41
3. Analysis Methodology 44
3.1. Quality Control Procedures 44
3.2. Overview of Item Response Theory (IRT) 48
3.2.1. Dichotomous Item Calibration 49
3.2.2. Polytomous Item Calibration 54
3.2.3. Calculation of IRT Scale Scores 57
3.2.4. Selection of an IRT Model 57
3.2.5. Evaluating Items Using Empirical Item Characteristic Curves (ICC) 59
3.2.6. Item Information and Measurement Precision 62
3.2.7. Item Response Theory Estimation Using PARSCALE 66
3.3. Construct Validity: Assessing Dimensionality 69
3.4. Group Differences in Item Functioning 70
3.5. Development of the Kindergarten Through Second-Grade Longitudinal Scale 74
3.5.1. Evaluating Common Item Functioning 74
3.5.2. Concurrent Calibration and Chain Linking 75
3.5.3. Computation of Final Scale Scores 78
4. Development of the Two-Stage Cognitive Assessment Test Forms 79
4.1. Development of the Item Pool 79
4.1.1. Field Test Design 80
4.1.2. Methods Used to Analyze Data for Design of the National Assessments 80
4.1.3. Criteria Guiding the Selection of Items for the National Assessments 84
4.2. Reading 87
4.2.1. Estimated Ability Levels for the ECLS-K:2011 National Samples and Target Ranges for Item Difficulties 88
4.3. Mathematics 95
4.3.1. Estimated Ability Levels for the ECLS-K:2011 National Samples and Target Ranges for Item Difficulties 95
4.4. Science 99
4.4.1. Estimated Ability Levels for the ECLS-K:2011 National Samples and Target Ranges for Item Difficulties 100
4.5. Cut Scores Used for Routing 104
4.5.1. Reading 106
4.5.2. Mathematics 106
4.5.3. Science 107
4.6. Discontinue Rules 107
4.6.1. Reading 108
4.6.2. Mathematics 109
4.6.3. Science 110
5. Psychometric Characteristics of the ECLS-K:2011 Direct Cognitive Battery 111
5.1. Routing of Children Through the National Assessments 111
5.1.1. Kindergarten Rounds 112
5.1.2. First-Grade Rounds 114
5.1.3. Second-Grade Rounds 117
5.2. Scoring the National Assessment 118
5.2.1. Confirmation of IRT Assumptions 118
5.2.2. Analysis of Differential Item Functioning (DIF) 121
5.2.2.1. Kindergarten Rounds 121
5.2.2.2. First-Grade Rounds 123
5.2.2.3. Second-Grade Rounds 124
5.2.3. Assessment Score Reliability 126
5.2.4. Item Response Theory (IRT)-Based Scores Developed for the ECLS-K:2011 127
5.2.4.1. Theta and the Standard Error of Measurement (SEM) of Theta 128
5.2.4.2. IRT Scale Scores 130
5.2.5. Raw Number-Right Scores for the ECLS-K:2011 131
5.3. Reading Assessment 131
5.3.1. Samples and Associated Statistics for the Kindergarten Rounds 131
5.3.2. Samples and Associated Statistics for the First-Grade Rounds 134
5.3.3. Samples and Associated Statistics for the Second-Grade Rounds 135
5.3.4. Score Statistics 136
5.3.5. Reliabilities 138
5.4. Spanish Early Reading Skills (SERS) 140
5.4.1. Samples and Associated Statistics for the Kindergarten Rounds 140
5.4.2. Samples and Associated Statistics for the First-Grade Rounds 140
5.4.3. Score Statistics 141
5.4.4. Reliabilities 143
5.5. Mathematics Assessment 144
5.5.1. Samples and Associated Statistics for the Kindergarten Rounds 144
5.5.2. Samples and Associated Statistics for the First-Grade Rounds 145
5.5.3. Samples and Associated Statistics for the Second-Grade Rounds 147
5.5.4. Score Statistics 148
5.5.5. Reliabilities 149
5.5.6. Comparability of the English and Spanish Mathematics Tests 150
5.6. Science Assessment 152
5.6.1. Samples and Associated Statistics in Spring Kindergarten 152
5.6.2. Samples and Associated Statistics in the First-Grade Rounds 153
5.6.3. Samples and Associated Statistics in the Second-Grade Rounds 155
5.6.4. Score Statistics 156
5.6.5. Reliabilities 157
5.7. Evaluating the Kindergarten Through Second-Grade Assessment Battery: Construct Validity and Measurement of Growth 158
5.7.1. Is the Difficulty of the Tests Suitable for Children's Ability Levels? 159
5.7.2. Do the Tests Measure the Right Content? 159
5.7.3. Do the Data Constitute a Cohesive Scale Suitable for Longitudinal Measurement? 160
5.7.3.1. Precision of Gain Scores Used to Model Growth 163
5.7.4. Relationship of the Cognitive Test Scores to Scores in Different Rounds and Different Subjects 163
5.8. Selection and Use of Scores 166
5.8.1. Choosing the Appropriate Score for Analysis 166
5.8.2. Analytic Considerations for Measuring Gains in the ECLS-K:2011 168
6. Psychometric Characteristics of the Executive Function Measures 170
6.1. Dimensional Change Card Sort (DCCS) 170
6.1.1. Mean Scores 181
6.1.2. Using the Computerized Version of the DCCS for Children Age 8 and Older 191
6.2. Numbers Reversed 193
6.2.1. Mean Scores 198
7. Psychometric Characteristics of the Indirect Measures 207
7.1. Teacher Measures 208
7.1.1. Children's Behavior Questionnaire (CBQ) and Temperament in Middle Childhood Questionnaire (TMCQ) 208
7.1.2. Teacher-Reported Social Skills 214
7.1.3. Teacher-Reported Approaches to Learning Items and Scale 224
7.1.4. Student-Teacher Relationship Scale 226
7.2. Parent Measures 230
7.2.1. Parent-Reported Social Skills 230
7.2.2. Parent-Reported Approaches to Learning Items and Scale 237
References 240
Appendix A. Plots of IRT Test Information Functions for Reading, Mathematics, and Science 245
Appendix B. IRT Estimation Using Parscale 247
Appendix C. ECLS-K:2011 Kindergarten, First-Grade, and Second-Grade Reading, Mathematics, and Science IRT Item Parameters 252
Appendix D. Plots of IRT Theta and Scale Score Distributions in Reading, Mathematics, and Science 261
Table 2-1. Reading content categories and target percentages of items, ECLS-K:2011 kindergarten through third-grade assessments 38
Table 2-2. Mathematics content categories and target percentages of items, ECLS-K:2011 kindergarten through third-grade assessments 38
Table 2-3. Science content categories and target percentages of items, ECLS-K:2011 kindergarten through third-grade assessments 39
Table 3-1. Comparison of one-, two-, and three-parameter IRT models of relative model fit (AIC and BIC) 58
Table 4-1. Means and standard deviations of estimated reading ability level (theta) for children in first grade 89
Table 4-2. Means and standard deviations of estimated reading ability level (theta) for children in second grade 89
Table 4-3. Peak difficulty ranges for the national first-grade reading assessment, routing plus second stage: ECLS-K:2011 90
Table 4-4. Peak difficulty ranges for the national second-grade reading assessment, routing plus second stage: ECLS-K:2011 91
Table 4-5. Framework targets and items by content area for the national kindergarten, first-grade, and second-grade reading assessments: ECLS-K:2011 92
Table 4-6. Dichotomous and polytomous items and score values for the national kindergarten, first-grade, and second-grade reading assessments: ECLS-K:2011 93
Table 4-7. Framework targets and items by content area for the national Spanish early reading skills (SERS) assessment: ECLS-K:2011 94
Table 4-8. Means and standard deviations of estimated mathematics ability level (theta) for children in first grade 96
Table 4-9. Means and standard deviations of estimated mathematics ability level (theta) for children in second grade 96
Table 4-10. Peak difficulty ranges for the national first-grade mathematics assessment, routing plus second stage: ECLS-K:2011 97
Table 4-11. Peak difficulty ranges for the national second-grade mathematics assessment, routing plus second stage: ECLS-K:2011 98
Table 4-12. Framework targets and items by content area for the national kindergarten, first-grade, and second-grade mathematics assessments: ECLS-K:2011 99
Table 4-13. Means and standard deviations of estimated science ability level (theta) for children in first grade 101
Table 4-14. Means and standard deviations of estimated science ability level (theta) for children in second grade 101
Table 4-15. Peak difficulty ranges for the national first-grade science assessment, routing plus second stage: ECLS-K:2011 102
Table 4-16. Peak difficulty ranges for the national second-grade science assessment, routing plus second stage: ECLS-K:2011 103
Table 4-17. Framework targets and items by content area for the national kindergarten, first-grade, and second-grade science assessments: ECLS-K:2011 104
Table 5-1. Component analysis percentages by component by domain, ECLS-K:2011 fall and spring kindergarten data collections: School year 2010-11 119
Table 5-2. Component analysis percentages by component by domain, ECLS-K:2011 fall and spring first-grade data collections: School year 2011-12 120
Table 5-3. Component analysis percentages by component by domain, ECLS-K:2011 fall and spring second-grade data collections: School year 2012-13 120
Table 5-4. Reading assessment differential item functioning, ECLS-K:2011 fall and spring kindergarten data collections: School year 2010-11 122
Table 5-5. Mathematics assessment differential item functioning, ECLS-K:2011 fall and spring kindergarten data collections: School year 2010-11 122
Table 5-6. Reading assessment differential item functioning, ECLS-K:2011 fall and spring first-grade data collections: School year 2011-12 123
Table 5-7. Mathematics assessment differential item functioning, ECLS-K:2011 fall and spring first-grade data collections: School year 2011-12 124
Table 5-8. Science assessment differential item functioning, ECLS-K:2011 fall and spring first-grade data collections: School year 2011-12 124
Table 5-9. Reading assessment differential item functioning, ECLS-K:2011 fall and spring second-grade data collections: School year 2012-13 125
Table 5-10. Mathematics assessment differential item functioning, ECLS-K:2011 fall and spring second-grade data collections: School year 2012-13 125
Table 5-11. Science assessment differential item functioning, ECLS-K:2011 fall and spring second-grade data collections: School year 2012-13 126
Table 5-12. Kindergarten reading assessment samples, ECLS-K:2011 fall and spring kindergarten data collections: School year 2010-11 133
Table 5-13. First-grade reading assessment samples, ECLS-K:2011 fall and spring first-grade data collections: School year 2011-12 135
Table 5-14. Second-grade reading assessment samples, ECLS-K:2011 fall and spring second-grade data collections: School year 2012-13 136
Table 5-15. Reading assessment statistics, by IRT-based score, ECLS-K:2011 fall and spring kindergarten, fall and spring first-grade, and fall and spring second-grade data collections... 137
Table 5-16. Reading assessment statistics, by raw number-right score, ECLS-K:2011 fall and spring kindergarten and fall and spring first-grade data collections: School years 2010-11 and 2011-12 138
Table 5-17. Reading assessment reliabilities, ECLS-K:2011 fall and spring kindergarten, fall and spring first-grade, and fall and spring second-grade data collections: School years 2010-11,... 139
Table 5-18. Kindergarten SERS assessment samples, ECLS-K:2011 fall and spring kindergarten data collections: School year 2010-11 140
Table 5-19. First-grade SERS assessment samples, ECLS-K:2011 fall and spring first-grade data collections: School year 2011-12 141
Table 5-20. SERS assessment statistics by IRT-based score, ECLS-K:2011 fall and spring kindergarten and fall and spring first-grade data collections: School years 2010-11 and 2011-12 142
Table 5-21. SERS and EBRS common item assessment statistics by raw number-right score, ECLS-K:2011 fall and spring kindergarten and fall and spring first-grade data collections... 143
Table 5-22. SERS assessment reliabilities, ECLS-K:2011 fall and spring kindergarten and fall and spring first-grade data collections: School years 2010-11 and 2011-12 144
Table 5-23. Kindergarten mathematics assessment samples, ECLS-K:2011 fall and spring kindergarten data collections: School year 2010-11 146
Table 5-24. First-grade mathematics assessment samples, ECLS-K:2011 fall and spring first-grade data collections: School year 2011-12 147
Table 5-25. Second-grade mathematics assessment samples, ECLS-K:2011 fall and spring second-grade data collections: School year 2012-13 148
Table 5-26. Mathematics assessment statistics by IRT-based score, ECLS-K:2011 fall and spring kindergarten, fall and spring first-grade, and fall and spring second-grade data collections... 149
Table 5-27. Mathematics assessment reliabilities, ECLS-K:2011 fall and spring kindergarten, fall and spring first-grade, and fall and spring second-grade data collections: School years 2010-11,... 150
Table 5-28. Kindergarten mathematics assessment samples, by language of assessment, ECLS-K:2011 fall and spring kindergarten data collections: School year 2010-11 151
Table 5-29. First-grade mathematics assessment samples, by language of assessment, ECLS-K:2011 fall and spring first-grade data collections: School year 2011-12 152
Table 5-30. Kindergarten science assessment sample: School year 2010-11 153
Table 5-31. First-grade science assessment samples, ECLS-K:2011 fall and spring first-grade data collections: School year 2011-12 155
Table 5-32. Second-grade science assessment samples, ECLS-K:2011 fall and spring second-grade data collections: School year 2012-13 156
Table 5-33. Science assessment statistics by IRT-based score, ECLS-K:2011 spring kindergarten, fall and spring first-grade, and fall and spring second-grade data collections: Spring 2011 and... 157
Table 5-34. Science assessment reliabilities, ECLS-K:2011 spring kindergarten, fall and spring first-grade, and fall and spring second-grade data collections: Spring 2011 and school years... 158
Table 5-35. Correlations of IRT theta score across rounds, by subject, ECLS-K:2011: School years 2010-11, 2011-12, and 2012-13 165
Table 5-36. Correlations of IRT theta score across subjects, by round, ECLS-K:2011: School years 2010-11, 2011-12, and 2012-13 166
Table 6-1. Percentage of round participants assigned each code on the kindergarten and first-grade completion flag variables for the physical version of the Dimensional Change Card Sort... 181
Table 6-2. Percentage of round participants assigned each code on the completion flag variables for the computerized version of the Dimensional Change Card Sort for fall and spring second... 181
Table 6-3. Dimensional Change Card Sort (physical version) variable names, descriptions, value ranges, weighted means, and standard deviations for fall and spring kindergarten and fall and... 183
Table 6-4. Mean Dimensional Change Card Sort (physical version) total score for fall and spring kindergarten, by data collection round and child characteristics: School year 2010-11 184
Table 6-5. Mean Dimensional Change Card Sort (physical version) pre-switch score, by data collection round and child characteristics: School year 2011-12 185
Table 6-6. Mean Dimensional Change Card Sort (physical version) post-switch score, by data collection round and child characteristics: School year 2011-12 186
Table 6-7. Mean Dimensional Change Card Sort (physical version) Border Game score, by data collection round and child characteristics: School year 2011-12 187
Table 6-8. Mean Dimensional Change Card Sort (physical version) total score, by data collection round and child characteristics: School year 2011-12 188
Table 6-9. Dimensional Change Card Sort (computerized version) variable names, descriptions, value ranges, weighted means, and standard deviations, second grade: School year 2012-13 189
Table 6-10. Mean Dimensional Change Card Sort (computerized version) total score, by data collection round and child characteristics: School year 2012-13 190
Table 6-11. Numbers Reversed variable names, descriptions, value ranges, weighted means, and standard deviations: School years 2010-11, 2011-12 and 2012-13 199
Table 6-12. Mean Numbers Reversed grade standard score for fall and spring kindergarten, by child characteristics: School year 2010-11 200
Table 6-13. Mean Numbers Reversed grade percentile rank for fall and spring kindergarten, by child characteristics: School year 2010-11 201
Table 6-14. Mean Numbers Reversed W-ability score, by data collection round and child characteristics: School years 2011-12 and 2012-13 202
Table 6-15. Mean Numbers Reversed age standard score, by data collection round and child characteristics: School years 2011-12 and 2012-13 203
Table 6-16. Mean Numbers Reversed grade standard score, by data collection round and child characteristics: School years 2011-12 and 2012-13 204
Table 6-17. Mean Numbers Reversed age percentile rank, by data collection round and child characteristics: School years 2011-12 and 2012-13 205
Table 6-18. Mean Numbers Reversed grade percentile rank, by data collection round and child characteristics: School years 2011-12 and 2012-13 206
Table 7-1. Children's Behavior Questionnaire variable names, descriptions, value ranges, weighted means, and standard deviations: Spring 2012 210
Table 7-2. Temperament in Middle Childhood Questionnaire variable names, descriptions, value ranges, weighted means, and standard deviations: Spring 2013 211
Table 7-3. Reliability estimates for the teacher-reported attentional focus and inhibitory control: Spring 2012 and spring 2013 211
Table 7-4. Intercorrelations among attentional focusing and inhibitory control scales in first and second grades: Spring 2012 and spring 2013 212
Table 7-5. Mean Children's Behavior Questionnaire attentional focus and inhibitory control scores, by child characteristics, spring first grade: Spring 2012 213
Table 7-6. Mean Temperament in Middle Childhood Questionnaire attentional focus and inhibitory control scores, by child characteristics, spring second grade: Spring 2013 214
Table 7-7. Teacher-reported social skills scales variable names, descriptions, value ranges, weighted means, and standard deviations: School years 2011-12 and 2012-13 216
Table 7-8. Reliability estimates for the teacher-reported social skills scales: School years 2011-12 and 2012-13 217
Table 7-9. Within-round correlations of teacher-reported social skills: School years 2010-11, 2011-12, and 2012-13 219
Table 7-10. Mean teacher-reported self-control score, by data collection round and child characteristics: School years 2011-12 and 2012-13 220
Table 7-11. Mean teacher-reported interpersonal skills score, by data collection round and child characteristics: School years 2011-12 and 2012-13 221
Table 7-12. Mean teacher-reported externalizing problem behaviors score, by data collection round and child characteristics: School years 2011-12 and 2012-13 222
Table 7-13. Mean teacher-reported internalizing problem behaviors score, by data collection round and child characteristics: School years 2011-12 and 2012-13 223
Table 7-14. Teacher-reported Approaches to Learning scale variable names, descriptions, value ranges, weighted means, and standard deviations: School years 2011-12 and 2012-13 224
Table 7-15. Within-round correlations of the teacher-reported Approaches to Learning scale and the teacher-reported social skills scales: School years 2010-11, 2011-12, and 2012-13 225
Table 7-16. Mean teacher-reported Approaches to Learning scale scores, by data collection round and child characteristics: School years 2011-12 and 2012-13 226
Table 7-17. Student-Teacher Relationship Scale variable names, descriptions, value ranges, weighted means, and standard deviations: Spring 2012 and spring 2013 227
Table 7-18. Reliability estimates for the Student-Teacher Relationship Scale teacher-reported closeness and conflict scores: Spring 2012 and spring 2013 228
Table 7-19. Mean Student-Teacher Relationship Scale teacher-reported closeness score, by data collection round and child characteristics: Spring 2012 and spring 2013 229
Table 7-20. Mean Student-Teacher Relationship Scale teacher-reported conflict score, by data collection round and child characteristics: Spring 2012 and spring 2013 230
Table 7-21. Parent-reported social skills scales variable names, descriptions, value ranges, weighted means, and standard deviations: Spring 2012 231
Table 7-22. Reliability estimates for the parent-reported social skills scales: Spring 2012 232
Table 7-23. Correlations of parent-reported social skills in the spring of first grade: Spring 2012 232
Table 7-24. Correlations between teacher-reported social skills and parent-reported social skills in the spring of first grade: Spring 2012 233
Table 7-25. Mean parent-reported self-control score, by child characteristics: Spring 2012 234
Table 7-26. Mean parent-reported social interaction score, by child characteristics: Spring 2012 235
Table 7-27. Mean parent-reported sad/lonely score, by child characteristics: Spring 2012 236
Table 7-28. Mean parent-reported impulsive/overactive score, by child characteristics: Spring 2012 237
Table 7-29. Parent-reported Approaches to Learning scale variable names, descriptions, value ranges, weighted means, and standard deviations: Spring 2012 238
Table 7-30. Mean parent-reported Approaches to Learning scale scores, by child characteristics: Spring 2012 239
Figure 3-1. Three-parameter IRT logistic function for a hypothetical dichotomous test item 50
Figure 3-2. Three-parameter IRT logistic functions for seven hypothetical dichotomous test items with different difficulty (b) 52
Figure 3-3. Three-parameter IRT logistic functions for two hypothetical dichotomous test items with different discrimination (a) 53
Figure 3-4. Three-parameter IRT logistic function for a hypothetical polytomous test item of a two-item testlet 55
Figure 3-5. Example of an empirical dichotomous item characteristic curve (ICC) for a well-functioning item: ECLS-K:2011 fall 2009 field test 60
Figure 3-6. Example of an empirical dichotomous item characteristic curve (ICC) for a poorly functioning item: ECLS-K:2011 fall 2009 field test 61
Figure 3-7. Item characteristic curve (ICC) compared to item information function (IIF) 63
Figure 3-8. Example test information function (TIF) 66
Exhibits
Exhibit 5-1. Routing path for the direct child assessment in the ECLS-K:2011 kindergarten year 113
Exhibit 5-2. Routing path for the direct child assessment in the ECLS-K:2011 first-grade year 116
Exhibit 5-3. Routing path for the direct child assessment in the ECLS-K:2011 second-grade year 117
Table C-1. ECLS-K:2011 kindergarten, first-grade, and second-grade reading IRT item parameters on the kindergarten scale: School years 2010-11, 2011-12, and 2012-13 252
Table C-2. ECLS-K:2011 kindergarten, first-grade, and second-grade mathematics IRT item parameters on the kindergarten scale: School years 2010-11, 2011-12, 2012-13 254
Table C-3. ECLS-K:2011 kindergarten, first-grade, and second-grade science IRT item parameters on the kindergarten scale: School years 2010-11, 2011-12, and 2012-13 258
Table C-4. ECLS-K kindergarten and first-grade Spanish early reading skills IRT item parameters on the concurrent kindergarten and first-grade scale: School years 2010-11 and 2011-12 260
Figure A-1. IRT test information function for reading for school years 2010-11, 2011-12, and 2012-13 245
Figure A-2. IRT test information function for mathematics for school years 2010-11, 2011-12, and 2012-13 245
Figure A-3. IRT test information function for science for school years 2010-11, 2011-12, and 2012-13 246
Figure D-1. IRT theta score distributions in reading for school years 2010-11, 2011-12, and 2012-13 261
Figure D-2. IRT scale score distributions in reading for school years 2010-11, 2011-12, and 2012-13 261
Figure D-3. IRT theta score distributions in mathematics for school years 2010-11, 2011-12, and 2012-13 262
Figure D-4. IRT scale score distributions in mathematics for school years 2010-11, 2011-12, and 2012-13 262
Figure D-5. IRT theta score distributions in science for school years 2010-11, 2011-12, and 2012-13 263
Figure D-6. IRT scale score distributions in science for school years 2010-11, 2011-12, and 2012-13 263