Title page
Contents
EXECUTIVE SUMMARY 4
Introduction 4
Purpose of This Report 4
Findings 4
CHAPTER 1: INTRODUCTION 11
1.1. Challenges for Conducting ABS Mail Surveys 11
1.1.1. Identifying Eligible Household Members at Sampled Addresses 12
1.1.2. Selecting One or More Target Respondents 12
1.1.3. Requesting Responses from Multiple Household Members 13
1.2. NATES Methods 14
CHAPTER 2: RESPONSE RATE, REPRESENTATIVENESS, AND RESPONSE QUALITY 16
2.1. Unit Response 16
2.1.1. Household-Level Response 17
2.1.2. Person-Level Response 18
2.1.3. Overall Response 22
2.2. Sample Representativeness 23
2.2.1. Comparison of Respondents and Eligible Sample on Frame Variables 23
2.2.2. Respondent Demographic Characteristics Compared to a Gold Standard 24
2.2.3. Key Estimates Compared to a Prior Administration 26
2.3. Response Quality 27
2.3.1. Item Nonresponse 27
2.3.2. Skip Errors of Omission 28
CHAPTER 3: SCREENER ITEM RESULTS 29
3.1. Screener Item Responses 29
3.2. Household-Level Item Nonresponse 30
3.3. Inconsistent Screener Item Responses 31
3.3.1. Within-Household Inconsistency 31
3.3.2. Inconsistency with Number of Eligible Questionnaires Returned 32
3.4. Editing of Screener Item Responses 33
CHAPTER 4: COMPOSITE-BOOKLET VERSUS INDIVIDUAL-BOOKLETS EXPERIMENT 34
4.1. Impact of Booklet Format on Unit Response 34
4.1.1. Within-Household Response 35
4.2. Impact of Booklet Format on Sample Representativeness 37
4.3. Impact of Booklet Format on Response Quality 38
CHAPTER 5: SUMMARY AND CONSIDERATIONS FOR THE FUTURE 39
5.1. Findings 39
5.2. Limitations 43
5.3. Considerations for Future Single-Phase, All-Adults Surveys 44
REFERENCES 46
APPENDIX A: NATIONAL ADULT TRAINING AND EDUCATION SURVEY QUESTIONNAIRE 48
APPENDIX B: SAMPLING, DATA COLLECTION, WEIGHTING, AND ESTIMATION PROCEDURES 66
B.1. Sampling 66
B.2. Data Collection 67
B.3. Weighting 68
B.4. Variance Estimation 78
B.5. Final Response Status and Household Size Determination 79
B.6. Response Rate Calculation 80
APPENDIX C: EDITING RULES FOR DETERMINING NUMBER OF ELIGIBLE HOUSEHOLD MEMBERS 83
APPENDIX D: ADDITIONAL TABLES 89
APPENDIX E: STANDARD ERROR TABLES 108
Table 2.1. Number of sampled NATES households, by response status: 2013 17
Table 2.2. Number of persons in responding NATES households, by response status: 2013 18
Table 2.3. Percentage distribution of responding households, by survey and number of eligible adults: 2013 19
Table 2.4. Percentage distribution of responding households with at least one eligible adult, by number of eligible, complete sets of topical item responses returned: 2013 20
Table 2.5. Percentage of responding households with at least one eligible adult, by proportion of eligible adults who responded and number of eligible adults in household: 2013 20
Table 2.6. Percentage of identical responses within responding households that returned at least two sets of eligible, complete topical item responses, by number of eligible, complete sets of topical item responses returned and selected items: 2013 22
Table 2.7. Percentage distribution of respondents, by survey and selected characteristics: 2013 25
Table 2.8. Percentage distribution of responses to selected key survey items, by survey and item: 2010 and 2013 27
Table 2.9. Item nonresponse rates in NATES, by key survey items: 2013 28
Table 2.10. Percentage of skip errors of omission in NATES, by key survey items: 2013 28
Table 3.1. Percentage distribution of responses to screener items, by item, booklet condition, and number of eligible adults: 2013 30
Table 3.2. Percentage of households with at least one eligible responding adult in which all respondents skipped one or both screener items, by booklet condition and item: 2013 31
Table 3.3. Rate of within-household inconsistency in screener information in individual-booklet households that returned at least two eligible, complete sets of topical item responses, by number of eligible questionnaires returned and screener item: 2013 32
Table 3.4. Percentage of screener item responses for which the reported number of eligible adults was less than the number of eligible, complete sets of topical item responses returned, by booklet condition: 2013 32
Table 3.5. Percentage of responding households in which screener item responses required editing, by booklet condition and reason for editing: 2013 33
Table 4.1. Overall response rate of surveyed NATES households and individuals, by booklet condition and response rate: 2013 35
Table 4.2. Percentage distribution of responding households, by booklet condition and number of eligible adults: 2013 36
Table 4.3. Percentage of responding households with at least one eligible adult, by booklet condition, proportion of eligible adults who responded, and number of eligible adults in household: 2013 37
Table B.1. Timeline of NATES data collection activities: 2013 67
Table B.2. Components of NATES household-level base weights: 2014 69
Table B.3. Variables used in NATES household-level CHAID analysis: 2013 70
Table B.4. NATES household-level nonresponse adjustment cells: 2013 72
Table B.5. Variables used in NATES person-level CHAID analysis: 2013 75
Table B.6. NATES person-level nonresponse adjustment cells: 2013 76
Table B.7. NATES household-level outcome codes, definition of each outcome code, and number of sampled households with each outcome code: 2013 81
Table B.8. NATES person-level outcomes: 2013 82
Table C.1. Editing rules for determining number of eligible household members 84
Table D.1. Summary of logistic regression analyses for variables predicting whether or not all eligible household members responded to the survey 89
Table D.2. Percentage distribution of NATES eligible sample and respondents, by selected frame characteristics: 2013 89
Table D.2A. Standard errors for Table D.2: Percentage distribution of NATES eligible sample and respondents, by selected frame characteristics: 2013 92
Table D.3. Overall number of sampled NATES households, by booklet condition and response status: 2013 94
Table D.4. Number of persons in responding NATES households, by booklet condition and response status: 2013 94
Table D.5. Percentage distribution of responding households with at least one eligible adult, by booklet condition and number of eligible, complete sets of topical item responses returned: 2013 95
Table D.5A. Standard errors for Table D.5: Percentage distribution of responding households with at least one eligible adult, by booklet condition and number of eligible, complete sets of topical item responses returned: 2013 95
Table D.6. Percentage distribution of NATES eligible sample and NATES respondents, by booklet condition and selected frame characteristics: 2013 96
Table D.6A. Standard errors for Table D.6: Percentage distribution of NATES eligible sample and NATES respondents, by booklet condition and selected frame characteristics: 2013 99
Table D.7. Percentage distribution of respondents, by survey, booklet condition, and selected characteristics: 2013 102
Table D.7A. Standard errors for Table D.7: Percentage distribution of respondents, by survey, booklet condition, and selected characteristics: 2013 103
Table D.8. Percentage distribution of responses to key items, by survey, booklet condition, and items: 2010 and 2013 104
Table D.8A. Standard errors for Table D.8: Percentage distribution of responses to key items, by survey, booklet condition, and items: 2010 and 2013 105
Table D.9. Item nonresponse rates in NATES, by booklet condition and key survey items: 2013 106
Table D.9A. Standard errors for Table D.9: Item nonresponse rates in NATES, by booklet condition and key survey items: 2013 106
Table D.10. Percentage of skip errors of omission in NATES, by booklet condition and key survey items: 2013 107
Table D.10A. Standard errors for Table D.10: Percentage of skip errors of omission in NATES, by booklet condition and key survey items: 2013 107
Table E.1. Standard errors for Table 2.3: Percentage distribution of responding households, by survey and number of eligible adults: 2013 108
Table E.2. Standard errors for Table 2.4: Percentage distribution of responding households with at least one eligible adult, by number of eligible, complete sets of topical item responses returned: 2013 108
Table E.3. Standard errors for Table 2.5: Percentage of responding households with at least one eligible adult, by proportion of eligible adults who responded and number of eligible adults in household: 2013 108
Table E.4. Standard errors for Table 2.6: Percentage of identical responses within responding households that returned at least two sets of eligible, complete topical item responses, by number of eligible,... 109
Table E.5. Standard errors for Table 2.7: Percentage distribution of respondents, by survey and selected characteristics: 2013 110
Table E.6. Standard errors for Table 2.8: Percentage distribution of responses to selected key survey items, by survey and item: 2010 and 2013 111
Table E.7. Standard errors for Table 2.9: Item nonresponse rates in NATES, by key survey items: 2013 111
Table E.8. Standard errors for Table 2.10: Percentage of skip errors of omission in NATES, by key survey items: 2013 112
Table E.9. Standard errors for Table 3.1: Percentage distribution of responses to screener items, by item, booklet condition, and number of eligible adults: 2013 112
Table E.10. Standard errors for Table 3.2: Percentage of households with at least one eligible responding adult in which all respondents skipped one or both screener items, by booklet condition and item: 2013 112
Table E.11. Standard errors for Table 3.3: Rate of within-household inconsistency in screener information in individual-booklet households that returned at least two eligible, complete sets of topical item responses,... 113
Table E.12. Standard errors for Table 3.4: Percentage of screener item responses for which the reported number of eligible adults was less than the number of eligible, complete sets of topical item responses returned,... 113
Table E.13. Standard errors for Table 3.5: Percentage of responding households in which screener item responses required editing, by booklet condition and reason for editing: 2013 113
Table E.14. Standard errors for Table 4.1: Overall response rate of surveyed NATES households and individuals, by booklet condition and response rate: 2013 114
Table E.15. Standard errors for Table 4.2: Percentage distribution of responding households, by booklet condition and number of eligible adults: 2013 114
Table E.16. Standard errors for Table 4.3: Percentage of responding households with at least one eligible adult, by booklet condition, proportion of eligible adults who responded, and number of eligible adults in household: 2013 114