Title page
Contents
1. Introduction 16
1.1. The Kindergarten Student Sample 17
1.2. Data Collection Instruments and Administration of Assessments 18
1.3. Contents of Report 19
2. Overview of The Content Coverage and Administration of the Direct Cognitive Assessment Instruments 21
2.1. Content Coverage of the Cognitive Assessments 21
2.1.1. Reading Test Specifications 23
2.1.1.1. Basic Reading Skills 23
2.1.1.2. Vocabulary 25
2.1.1.3. Comprehension 26
2.1.1.4. Continuity Between the ECLS-K and the ECLS-K:2011 Reading Frameworks 27
2.1.2. Mathematics Test Specifications 27
2.1.2.1. Number Properties and Operations 28
2.1.2.2. Measurement 29
2.1.2.3. Geometry 29
2.1.2.4. Data Analysis and Probability 29
2.1.2.5. Algebra 30
2.1.3. Science Test Specifications 30
2.1.3.1. Scientific Inquiry 31
2.1.3.2. Physical Science 31
2.1.3.3. Life Science 31
2.1.3.4. Earth and Space Science 32
2.2. Item and Time Allocation Across and Within Subject Areas 32
2.3. Mode of Administration 35
2.4. Inclusion of Children With Diverse Language Backgrounds and Language of Administration 37
3. Analysis Methodology 40
3.1. Quality Control Procedures 40
3.2. Overview of Item Response Theory (IRT) 43
3.2.1. Calculation of Scores and Treatment of Missing Data 49
3.2.2. Selection of an IRT Model 49
3.2.3. Evaluating Items Using Empirical Item Characteristic Curves (ICC) 50
3.2.4. Item Information and Measurement Precision 53
3.2.5. Item Response Theory Estimation Using PARSCALE 57
3.3. Construct Validity: Assessing Dimensionality 59
3.4. Group Differences in Item Functioning 60
3.5. Evaluating Common Item Functioning During the Development of the Kindergarten Longitudinal Scale 64
3.5.1. Concurrent Calibration and Computation of Final Scores 65
4. Development of the Two-Stage Cognitive Assessment Test Forms 67
4.1. Development of the Item Pool 67
4.2. Field Test Design 70
4.2.1. English Field Test Design 71
4.2.1.1. Reading Field Test Forms and Items 72
4.2.1.2. Mathematics Field Test Forms and Items 73
4.2.1.3. Science Field Test Forms and Items 73
4.2.2. Spanish Field Test Design 73
4.2.2.1. English Basic Reading Skills (EBRS) and Spanish Early Reading Skills (SERS) Field Test Items 74
4.3. Field Test Results and the Development of the National Assessments for the Kindergarten Data Collection 75
4.3.1. Methods Used to Analyze the Field Test Data 75
4.3.2. Criteria Guiding the Selection of Items for the National Kindergarten Assessments 78
4.3.2.1. Estimated Ability Levels for the ECLS-K:2011 Kindergarten National Sample and Target Ranges for Item Difficulties 79
4.3.2.2. Item Quality and Reliability 82
4.3.3. Composition of the Final National Kindergarten Assessments 83
4.3.3.1. Reading 83
4.3.3.2. Mathematics 91
4.3.3.3. Science 93
4.3.4. Performance Simulations and Cut Scores Used for Routing 96
5. Psychometric Characteristics of the ECLS-K:2011 Direct Cognitive Battery 99
5.1. Routing of Children Through the National Assessment 99
5.2. Scoring the National Assessment 101
5.2.1. Confirmation of Unidimensionality 102
5.2.2. Analysis of Differential Item Functioning (DIF) 103
5.2.3. Assessment Score Reliability 105
5.2.4. Item ResponseTheory (IRT)-Based Scores Developed for the ECLS-K:2011 105
5.2.4.1. Theta and the Standard Error of Measurement (SEM) of Theta 106
5.2.4.2. IRT Scale Scores 107
5.2.5. Raw Number-Right Scores for the ECLS-K:2011 108
5.3. Reading Assessment 109
5.3.1. Samples and Associated Statistics 109
5.3.2. Score Statistics 110
5.3.3. Reliabilities 112
5.4. Spanish Early Reading Skills (SERS) 112
5.4.1. Samples and Associated Statistics 112
5.4.2. Score Statistics 113
5.4.3. Reliabilities 114
5.5. Mathematics Assessment 115
5.5.1. Samples and Associated Statistics 115
5.5.2. Score Statistics 116
5.5.3. Reliabilities 117
5.5.4. Comparability of Spanish Mathematics Test 118
5.6. Science Assessment 119
5.6.1. Samples and Associated Statistics 119
5.6.2. Score Statistics 120
5.6.3. Reliabilities 120
5.7. Applications 121
5.7.1. Choosing the Appropriate Score for Analysis 121
5.7.2. Analytic Considerations for Measuring Gains in the ECLS-K:2011 123
6. Psychometric Characteristics of the Executive Function Measures 125
6.1. Dimensional Change Card Sort (DCCS) 125
6.1.1. Mean Scores 126
6.2. Numbers Reversed 129
6.2.1. Mean Scores 134
7. Psychometric Characteristics of the Indirect Measures 138
7.1. Teacher Measures 138
7.1.1. Children's Behavior Questionnaire (CBQ) 138
7.1.2. Teacher-Reported Social Skills 141
7.1.3. Teacher-Reported Approaches to Learning Items and Scale 148
7.1.4. Student-Teacher Relationship Scale 150
7.2. Parent Measures 153
7.2.1. Parent-Reported Social Skills 153
7.2.2. Parent-Reported Approaches to Learning Items and Scale 159
References 162
Appendix A. ECLS-K:2011 Fall 2009 Field Test Report 165
1. Introduction and Background 175
1.1. Study Overview 175
1.2. Design of ECLS-K:2011 Survey Instruments and Assessments 176
1.3. Purpose of Field Tests 177
1.3.1. Purpose of the Fall 2009 English Field Test (EFT) 178
1.3.2. Purpose of the Fall 2009 Spanish Field Test (SFT) 179
1.4. Field Test Data Collection 179
1.4.1. EFT Data Collection 180
1.4.2. SFT Data Collection 181
2. Field Test Sample Design 183
2.1. Field Test Samples 183
2.2. EFT Sample 183
2.3. SFT Sample 185
3. Data Collection 189
3.1. Field Organization 189
3.2. Field Staff Training 190
3.2.1. School Recruiter Training 190
3.2.2. Team Leader Training 190
3.2.3. Assessor Training 191
3.2.4. Health Technician Training 192
3.3. School Recruitment 193
3.4. Preassessment Call to the School Coordinator 194
3.5. Meeting With the School Coordinator 194
3.6. Assessment Logistics 194
3.7. Student and Teacher Identification 195
3.8. Conducting the Assessment 195
3.9. Hearing and Vision Screenings 198
3.10. Production Results for Child Assessments 201
3.10.1. English Field Test 201
3.10.2. Spanish Field Test 203
3.11. Production Results for Science Academic Rating Scales 204
3.12. Field Staff Communication 207
3.13. Quality Assurance 207
4. Assessor Feedback and Recommendations 209
4.1. Summary of Diaries and Debriefings 209
4.2. Recommendations on Specific Aspects of the Assessment 209
4.2.1. Assessment Time Frames 209
4.2.2. Overall Child Reactions 210
4.2.3. EFT Cognitive Assessment Items 210
4.2.4. Vision and Hearing Screening 211
4.2.5. Spanish Field Test 212
4.3. Assessors' Recommendations for Training 213
5. Field Test Analysis and Development of the Kindergarten Direct Cognitive Assessments 215
5.1. Field Test Designs and Item Pools 215
5.1.1. English Field Test and Item Pool 215
5.1.2. Spanish Field Test and Item Pool 218
5.2. Field Test Psychometric Analysis 218
5.2.1. Methodology 219
5.2.2. Analysis Results 223
5.2.2.1. English Field Test 225
5.2.2.2. Spanish Field Test 227
5.3. Design of the Kindergarten Tests 228
5.3.1. Reading and EBRS 231
5.3.2. Mathematics 237
5.3.3. Science 240
5.3.4. Spanish Early Reading Skills 243
5.4. Assessment Form Simulations and Review 246
5.4.1. Assessment Forms and Simulations 246
5.4.2. Expert Review of the Assessments 249
5.4.3. Sensitivity Review 250
6. Science Academic Rating Scale 251
6.1. Kindergarten 251
6.2. First Grade 253
6.3. Second Grade 256
6.4. Summary and Recommendations 258
Appendix B. IRT Estimation Using Parscale 260
Appendix C. ECLS-K:2011 Kindergarten Irt Item Parameters 266
Table 2-1. Reading content categories and target percentages of items, ECLS-K:2011 kindergarten through third grade assessments 33
Table 2-2. Mathematics content categories and target percentages of items, ECLS-K:2011 kindergarten through third grade assessments 34
Table 2-3. Science content categories and target percentages of items, ECLS-K:2011 kindergarten through third grade assessments 35
Table 4-1. Organization of booklets: ECLS-K:2011 fall 2009 field test 72
Table 4-2. Estimated means and standard deviations of reading ability level (theta) for children in kindergarten 85
Table 4-3. Peak difficulty ranges for the national kindergarten reading assessment, routing plus second stage: ECLS-K:2011 85
Table 4-4. Framework targets and items by content area for the national kindergarten reading assessment: ECLS-K:2011 87
Table 4-5. Subcategories of basic skills items included in the national kindergarten reading assessment: ECLS-K:2011 88
Table 4-6. Number of items in the national kindergarten Spanish early reading skills (SERS) assessment, by difficulty range: ECLS-K:2011 89
Table 4-7. Framework targets and items by content area for the national kindergarten Spanish early reading skills (SERS) assessment: ECLS-K:2011 90
Table 4-8. Subcategories of basic skills items included in the national kindergarten Spanish early reading skills (SERS) assessment: ECLS-K:2011 90
Table 4-9. Estimated means and standard deviations of mathematics ability level (theta) for children in kindergarten 91
Table 4-10. Peak difficulty ranges for the national kindergarten mathematics assessment, routing plus second stage: ECLS-K:2011 92
Table 4-11. Framework targets and items by content area for the national kindergarten mathematics assessment: ECLS-K:2011 93
Table 4-12. Estimated means and standard deviations of science ability level (theta) for children in the spring of kindergarten 95
Table 4-13. Framework targets and items by content area for the national kindergarten science assessment: ECLS-K:2011 96
Table 5-1. Component analysis percentages by component by domain, ECLS-K:2011 fall and spring kindergarten data collections: School year 2010-11 102
Table 5-2. Reading assessment differential item functioning, ECLS-K:2011 fall and spring kindergarten data collections: School year 2010-11 104
Table 5-3. Mathematics assessment differential item functioning, ECLS-K:2011 fall and spring kindergarten data collections: School year 2010-11 104
Table 5-4. Kindergarten reading assessment samples, ECLS-K:2011 fall and spring kindergarten data collections: School year 2010-11 110
Table 5-6. Reading assessment statistics, by raw number-right score, ECLS-K:2011 fall and spring kindergarten data collections: School year 2010-11 111
Table 5-7. Reading assessment reliabilities, ECLS-K:2011 fall and spring kindergarten data collections: School year 2010-11 112
Table 5-8. Kindergarten SERS assessment samples, ECLS-K:2011 fall and spring kindergarten data collections: School year 2010-11 113
Table 5-5. Reading assessment statistics, by IRT-based score, ECLS-K:2011 fall and spring kindergarten data collections: School year 2010-11 111
Table 5-9. Kindergarten SERS assessment statistics, by IRT-based score, ECLS-K:2011 fall and spring kindergarten data collections: School year 2010-11 114
Table 5-10. Kindergarten SERS assessment statistics, by raw number-right score, ECLS-K:2011 fall and spring kindergarten data collections: School year 2010-11 114
Table 5-11. Kindergarten SERS assessment reliabilities, ECLS-K:2011 fall and spring kindergarten data collections: School year 2010-11 115
Table 5-12. Kindergarten mathematics assessment samples, ECLS-K:2011 fall and spring kindergarten data collections: School year 2010-11 116
Table 5-13. Mathematics assessment statistics, by IRT-based score, ECLS-K:2011 fall and spring kindergarten data collections: School year 2010-11 117
Table 5-14. Mathematics assessment reliabilities, ECLS-K:2011 fall and spring kindergarten data collections: School year 2010-11 117
Table 5-15. Kindergarten mathematics assessment samples, by language of assessment, ECLS-K:2011 fall and spring kindergarten data collections: School year 2010-11 119
Table 5-16. Kindergarten science assessment sample: Spring 2011 120
Table 5-17. Science assessment statistics, by IRT-based score, ECLS-K:2011 spring kindergarten data collection: Spring 2011 120
Table 5-18. Science assessment reliability, ECLS-K:2011 spring kindergarten data collection: Spring 2011 121
Table 6-1. Dimensional Change Card Sort variable names, descriptions, value ranges, weighted means, and standard deviations: School year 2010-11 127
Table 6-2. Mean Dimensional Change Card Sort post-switch score, by data collection round and child characteristics: School year 2010-11 128
Table 6-3. Mean Dimensional Change Card Sort Border Game score, by data collection round and child characteristics: School year 2010-11 129
Table 6-4. Numbers Reversed variable names, descriptions, value ranges, weighted means, and standard deviations: School year 2010-11 134
Table 6-5. Mean Numbers Reversed W-ability score, by data collection round and child characteristics: School year 2010-11 135
Table 6-6. Mean Numbers Reversed standard score, by data collection round and child characteristics: School year 2010-11 136
Table 6-7. Mean Numbers Reversed percentile rank, by data collection round and child characteristics: School year 2010-11 137
Table 7-1. Children's Behavior Questionnaire variable names, descriptions, value ranges, weighted means, and standard deviations: School year 2010-11 139
Table 7-2. Mean Children's Behavior Questionnaire attentional focus score, by data collection round and child characteristics: School year 2010-11 140
Table 7-3. Mean Children's Behavior Questionnaire inhibitory control score, by data collection round and child characteristics: School year 2010-11 141
Table 7-4. Teacher-reported social skills scales variable names, descriptions, value ranges, weighted means, and standard deviations: School year 2010-11 142
Table 7-5. Reliability estimates for the teacher-reported social skills scales: School year 2010-11 143
Table 7-6. Eigenvalues and proportion of variance accounted for by the three factors extracted in principal components factor analysis with fall kindergarten teacher-reported social skills data: Fall 2010 144
Table 7-7. Eigenvalues and proportion of variance accounted for by the three factors extracted in principal components factor analysis with spring kindergarten teacher-reported social skills data: Spring 2011 144
Table 7-8. Mean teacher-reported self-control score, by data collection round and child characteristics: School year 2010-11 145
Table 7-9. Mean teacher-reported interpersonal skills score, by data collection round and child characteristics: School year 2010-11 146
Table 7-10. Mean teacher-reported externalizing problem behaviors score, by data collection round and child characteristics: School year 2010-11 147
Table 7-11. Mean teacher-reported internalizing problem behaviors score, by data collection round and child characteristics: School year 2010-11 148
Table 7-12. Teacher-reported Approaches to Learning Scale variable names, descriptions, value ranges, weighted means, and standard deviations: School year 2010-11 149
Table 7-13. Mean teacher-reported Approaches to Learning Scale scores, by data collection round and child characteristics: School year 2010-11 150
Table 7-14. Student-Teacher Relationship Scale variable names, descriptions, value ranges, weighted means, and standard deviations: Spring 2011 151
Table 7-15. Mean Student-Teacher Relationship Scale teacher-reported closeness score, by child characteristics: Spring 2011 152
Table 7-16. Mean Student-Teacher Relationship Scale teacher-reported conflict score, by child characteristics: Spring 2011 153
Table 7-17. Parent-reported social skills scales variable names, descriptions, value ranges, weighted means, and standard deviations: School year 2010-11 154
Table 7-18. Reliability estimates for the parent-reported social skills scales: School year 2010-11 155
Table 7-19. Eigenvalues and proportion of variance accounted for by the three factors extracted in principal components factor analysis with fall kindergarten parent-reported social skills data: Fall 2010 155
Table 7-20. Eigenvalues and proportion of variance accounted for by the three factors extracted in principal components factor analysis with spring kindergarten parent-reported social skills data: Spring 2011 156
Table 7-21. Mean parent-reported self-control score, by data collection round and child characteristics: School year 2010-11 156
Table 7-22. Mean parent-reported social interaction score, by data collection round and child characteristics: School year 2010-11 157
Table 7-23. Mean parent-reported sad/lonely score, by data collection round and child characteristics: School year 2010-11 158
Table 7-24. Mean parent-reported impulsive/overactive score, by data collection round and child characteristics: School year 2010-11 159
Table 7-25. Parent-reported Approaches to Learning Scale variable names, descriptions, value ranges, weighted means, and standard deviations: School year 2010-11 160
Table 7-26. Mean parent-reported Approaches to Learning Scale scores, by data collection round and child characteristics: School year 2010-11 161
Figure 3-1. Three-parameter IRT logistic function for a hypothetical test item 46
Figure 3-2. Three-parameter IRT logistic functions for seven hypothetical test items with different difficulty (b) 47
Figure 3-3. Three-parameter IRT logistic functions for two hypothetical test items with different discrimination (a) 48
Figure 3-4. Example of an empirical item characteristic curve (ICC) for a well-functioning item: ECLS-K:2011 fall 2009 field test 51
Figure 3-5. Example of an empirical item characteristic curve (ICC) for a poorly functioning item: ECLS-K:2011 fall 2009 field test 52
Figure 3-6. Item characteristic curve (ICC) compared to item information function (IIF) 54
Figure 3-7. Example test information function (TIF) 57
Exhibits
Exhibit 5-1. Routing path for the direct child assessment in the ECLS-K:2011 kindergarten year 100
Appendix A. ECLS-K:2011 FALL 2009 FIELD TEST REPORT 170
Table 1-1. Fall 2009 field test activities and schedule 179
Table 2-1. Characteristics of schools participating in the fall 2009 English field test 184
Table 2-2. Type of consent required for the fall 2009 English field test, by school type 184
Table 2-3. Characteristics of children participating in the fall 2009 English field test 185
Table 2-4. Characteristics of schools participating in the fall 2009 Spanish field test 186
Table 2-5. Characteristics of children participating in the fall 2009 Spanish field test 187
Table 3-1. Fall 2009 English field test work areas 189
Table 3-2. Fall 2009 Spanish field test work areas 189
Table 3-3. Components of the fall 2009 English field test assessment easels 197
Table 3-4. English field test: Number of direct assessments completed, by school characteristics: 2009 201
Table 3-5. Number of hearing screenings completed, by school characteristics: 2009 202
Table 3-6. Number of vision screenings completed, by school characteristics: 2009 203
Table 3-7. Spanish field test: Number of direct assessments completed, by school characteristics: 2009 204
Table 3-8. Number of teacher rating forms completed, by school characteristics: 2009 206
Table 5-1. Organization of booklets: 2009 216
Table 5-2. Example of item analysis tables: 2009 field test 220
Table 5-3. ECLS-K proficiency levels in reading, through third grade 230
Table 5-4. ECLS-K proficiency levels in mathematics, through third grade 230
Table 5-5. Estimated means and standard deviations of theta for kindergarten: Reading 231
Table 5-6. Peak and full difficulty ranges, routing plus second stage: Reading 233
Table 5-7. Framework targets and items by content area: Reading 234
Table 5-8. Subcategories of basic skills items in proposed pool: Reading 236
Table 5-9. Number of items overlapping across forms: Reading 237
Table 5-10. Number of items in proposed kindergarten assessment overlapping with the ECLS-K: Reading 237
Table 5-11. Estimated means and standard deviations of theta for kindergarten: Mathematics 238
Table 5-12. Peak and full difficulty ranges, routing plus second stage: Mathematics 238
Table 5-13. Framework targets and items by content area: Mathematics 239
Table 5-14. Number of items overlapping across forms: Mathematics 240
Table 5-15. Number of items in proposed kindergarten assessment overlapping with the ECLS-K: Mathematics 240
Table 5-16. Estimated means and standard deviations of theta for spring kindergarten: Science 242
Table 5-17. Framework targets and items by content area: Science 243
Table 5-18. Number of items in proposed kindergarten assessment overlapping with the ECLS-K: Science 243
Table 5-19. Number of items in proposed kindergarten assessment by difficulty range, SERS 244
Table 5-20. Framework targets and items by content area: SERS 245
Table 5-21. Subcategories of basic skills items in proposed pool: SERS 245
Table 5-22. Number of items in proposed kindergarten assessment overlapping with the ECLS-K: SERS 246
Table 5-23. Cutscores for the ECLS-K:2011 kindergarten assessment in mathematics 248
Table 5-24. Cutscores for the ECLS-K:2011 kindergarten assessment in reading: Router 1 249
Table 5-25. Cutscores for the ECLS-K:2011 kindergarten assessment in Reading: Router 2 249
Table 6-1. Average item scores across the completed kindergarten Science ARS forms and percent where the highest possible rating, the lowest possible rating, and not applicable... 252
Table 6-2. Average item scores across the completed first-grade Science ARS forms and percent where the highest possible rating, the lowest possible rating, and not applicable or skill... 255
Table 6-3. Average item scores across the completed second-grade Science ARS forms and percent where the highest possible rating, the lowest possible rating, and not applicable... 257
Appendix C. ECLS-K:2011 KINDERGARTEN IRT ITEM PARAMETERS 266
Table C-1. ECLS-K:2011 Kindergarten reading IRT item parameters: School year 2010-11 266
Table C-2. ECLS-K:2011 kindergarten mathematics IRT item parameters: School year 2010-11 269
Table C-3. ECLS-K:2011 kindergarten science IRT item parameters: School year 2010-11 272
Table C-4. ECLS-K kindergarten Spanish early reading skills IRT item parameters: School year 2010-11 273
Figure 5-1. Examples of IRT plots, ECLS-K:2011 fall 2009 field test 222
Appendix Exhibits
Exhibit 3-1. Assessments and hearing/vision exam schedule 199
Exhibit 3-2. Response scale for the Science Academic Rating Scale 205