Title page
Contents
1. Overview of PISA 2018 11
1.1. Introduction 11
1.2. What's New in PISA 2018? 12
1.3. What PISA Measures 14
1.4. PISA 2018 Administration 15
1.5. Organization of This Document 15
2. Sampling 17
2.1. International Requirements 17
2.2. School Sampling in the United States 18
2.2.1. School Sampling Frame 18
2.2.1.a. Enrollment of PISA-Eligible Students (ENR) 19
2.2.1.b. School Exclusions 20
2.2.1.c. Stratification 21
2.2.2. School Sample 23
2.2.2.a. Measure of Size and Small Schools 24
2.2.2.b. Substitute Schools 25
2.2.2.c. Tabulations Within Subgroups for Frame and Sample 26
2.3. Student Sampling in the United States 27
2.4. Teacher Sampling in the United States 28
3. School and Student Recruitment 29
3.1. Overview 29
3.2. Recruitment Training 31
3.3. Recruitment of Schools 31
3.3.1. Contacting States, Districts, and Schools 33
3.3.1.a. State Contact 33
3.3.1.b. District Contact 33
3.3.1.c. Special Handling Districts 34
3.3.1.d. Initial School Contact 34
3.3.2. Reasons for School Refusal to Participate 34
3.3.3. Solutions and Approaches Used with Refusing Schools 35
3.3.4. Final Results of School Recruitment 36
3.4. Student Recruitment 37
4. Instrument Development and Distribution 38
4.1. Test Instrument Design 38
4.2. Assessment Materials Development 42
4.3. Preparation of Instruments 43
4.4. Packaging and Distribution of Materials to Field Staff 43
5. Field Operations 44
5.1. Preassessment Contacts with School Staff 45
5.2. Data Collection Training 47
5.3. Data Collection Approach 48
5.4. Data Collection Activities 50
6. Response Rates 51
6.1. Participation Rates for the U.S. Schools, Students, and Teachers 51
6.1.1. School Participation 52
6.1.2. Student Participation 53
6.1.3. Teacher Participation 53
6.2. Excluded and Ineligible Schools and Students 53
6.2.1. School Exclusions and Ineligibility 53
6.2.2. Within-School Exclusions and Ineligibility 54
6.2.3. Exclusions and Ineligibility Rates in the United States 54
6.3. Participation Rates for All Education Systems 55
7. Data Management 56
7.1. Occupational Coding 56
7.2. Coding Open-Ended Student Responses 57
7.2.1. Lead Coder Training 58
7.2.2. Coder Training/Coding 59
7.2.3. Open Ended Coding System (OECS) 60
7.3. Data Editing and File Delivery 61
8. Processing, Scaling, and Weighting 63
8.1. International Data File Cleaning and Editing 63
8.2. Missing Data 63
8.3. Weights for U.S. Data 64
8.4. Scaling of Student Test Data 65
9. The PISA 2018 Data 66
9.1. National and International Variables 67
9.2. PISA 2018 U.S. International Datasets 67
9.3. U.S. National Data Files 69
9.4. Variable Names 74
9.5. Derived Variables 74
9.6. U.S. National Restricted-Use Variables 74
10. Using the PISA 2018 Data Files 76
10.1. Special Considerations-Plausible Values and Replicate Weights 76
10.2. Nonresponse Bias 79
10.3. Merging School, Student, and Teacher Data 79
References 81
Appendix A. PISA 2018 School Recruiting Materials 82
Appendix B. Student and Parent Materials 98
Appendix C. PISA 2018 School, Teacher, and Student Questionnaires 111
Appendix D. PISA 2018 Study Forms 112
Appendix E. Training Agendas 117
Appendix F. Variables Deleted from the U.S. PISA 2018 Questionnaires 121
Appendix G. PISA 2018 Inter-rater Item Reliability 127
Appendix H. Selected Indices from OECD and U.S. Composites 131
Appendix I. PISA 2018 Nonresponse Bias Analysis Report 132
Table 1. Percentage of PISA-eligible students in 2015 and student enrollment, number, and percentage of PISA-eligible students in PISA 2018, by grade 20
Table 2. Number and percentage of students and schools included in the PISA U.S. school sampling frame, by region: 2018 22
Table 3. Number and percentage of students and schools included in the PISA U.S. school sampling frame, by school type: 2018 22
Table 4. Number and percentage of students and schools included in the PISA U.S. school sampling frame, by region and school type: 2018 22
Table 5. Number of schools in the PISA U.S. school sample, by sampling stratum: 2018 23
Table 6. Estimated student yield and estimation parameters for PISA U.S. school sample, by sampling stratum: 2018 24
Table 7. Number and percentage of students and schools included in the PISA U.S. school sample, by Census region: 2018 26
Table 8. Number and percentage of students and schools included in the PISA U.S. school sample, by school type: 2018 26
Table 9. Number and percentage of students and schools included in the PISA U.S. school sample, by region and school type: 2018 27
Table 10. Summary of U.S. PISA school recruitment activities and outcomes: 2018 30
Table 11. PISA 2018 school type, by response status 36
Table 12. PISA 2018 participating schools, by type of parental consent 37
Table 13. Number of new and trend items in PISA 2018, by domain 38
Table 14. PISA U.S. schools, by response status: 2018 52
Table 15. Variables used only in the United States: 2018 72
Figure 1. PISA subjects assessments by year 13
Figure 2. PISA 2018 original and total school participation, by week 36
Exhibits
Exhibit 1. Timing of PISA 2018 CBA sessions 49
Exhibit 2. Timing of PISA 2018 CBA UH sessions 50
Exhibit 3. PISA 2018 OECS coding designs 58
Table F-1. Variables not used in the United States: 2018 121
Table G-1. PISA 2018 inter-rater item reliabilities 127
Table I-1. Selected characteristics for the nonresponse bias analysis of the U.S. PISA final school sample: 2018 134
Table I-2. Student and teacher participation rates after replacement of the U.S. PISA final school sample: 2018 134
Table I-3. Percentage distribution of eligible and participating schools in the U.S. PISA original sample, by selected school characteristics: 2018 141
Table I-4. Mean values of various characteristics for eligible and participating schools in the U.S. PISA original sample: 2018 142
Table I-5. Mean percentage of students eligible for free or reduced-price lunch, in eligible and participating public schools in the U.S. PISA original sample: 2018 143
Table I-6a. Logistic regression model parameters (with six race/ethnicity variables) using the U.S. PISA original school sample: 2018 145
Table I-6b. Logistic regression model parameters (with summed race/ethnicity percentage) using the U.S. PISA original school sample: 2018 146
Table I-6c. Logistic regression model parameters (with summed race/ethnicity percentage) using the U.S. PISA original public school sample: 2018 147
Table I-7. Percentage distribution of eligible and participating schools in the U.S. PISA final sample, by selected school characteristics: 2018 149
Table I-8. Mean values of various characteristics for eligible and participating schools in the U.S. PISA final sample: 2018 150
Table I-9. Mean percentage of students eligible for free or reduced-price lunch, in eligible and participating public schools in the U.S. PISA final sample: 2018 151
Table I-10a. Logistic regression model parameters (with six race/ethnicity variables) using the U.S. PISA final school sample: 2018 152
Table I-10b. Logistic regression model parameters (with summed race/ethnicity percentage) using the U.S. PISA final school sample: 2018 153
Table I-10c. Logistic regression model parameters (with summed race/ethnicity percentage) using the U.S. PISA final public school sample: 2018 154
Table I-11. Percentage distribution of eligible and participating schools in the U.S. PISA nonresponse-adjusted sample, by selected school characteristics: 2018 156
Table I-12. Mean values of various characteristics for eligible and participating schools in the U.S. PISA nonresponse-adjusted sample: 2018 157
Table I-13. Mean percentage of students eligible for free or reduced-price lunch, in eligible and participating public schools in the U.S. PISA nonresponse-adjusted sample: 2018 158
Table I-14. Characteristics with p values less than 0.05 and absolute relative bias greater than 10 percent, U.S. PISA schools: 2018 160
Appendix Exhibits
Exhibit A-1. State PISA 2018 letter from the NCES Commissioner 82
Exhibit A-2. School district PISA 2018 letter from the NCES Commissioner 83
Exhibit A-3. Public school principal 2018 letter from the NCES Commissioner 84
Exhibit A-4. Private school principal 2018 letter from the NCES Commissioner 85
Exhibit A-5. PISA 2018 study brochure and schedule of activities 86
Exhibit A-6. PISA 2018 Public school FAQ 90
Exhibit A-7. PISA 2018 Private school FAQ 92
Exhibit A-8. PISA 2018 Public school Teacher FAQ 94
Exhibit A-9. PISA 2018 Private school Teacher FAQ 96
Exhibit B-1. PISA 2018 Explicit parent materials 98
Exhibit B-2. PISA 2018 Implicit parent materials 100
Exhibit B-3. PISA 2018 Parent notification 102
Exhibit B-4. PISA 2085 Fact sheet for parents 103
Exhibit B-5. PISA 2018 Student invitation 105
Exhibit B-6. PISA 2018 Public school Student FAQ 106
Exhibit B-7. PISA 2018 Private school Student FAQ 108
Exhibit B-8. PISA 2018 Student certificate of volunteer service 110
Exhibit D-1. PISA 2018 Student Tracking Form 112
Exhibit D-2. PISA 2018 Session Attendance Form 113
Exhibit D-3. PISA 2018 Student Payment Receipt Form 114
Exhibit D-4. PISA 2018 Student Login Form 115
Exhibit D-5. PISA 2018 Teacher Tracking Form 116
Exhibit E-1. Test administrator training agenda 117
Exhibit E-2. Assistant administrator training agenda 120