Title Page
Abstract
Contents
ABBREVIATION 22
Chapter 1. Introduction 23
1.1. Background 23
1.2. Research Gaps & Rationale 25
1.4. Aims & Research Questions 28
1.5. Roadmap of the thesis 29
Chapter 2. Literature Review 30
2.1. The assessment of listening comprehension 30
2.1.1. The characteristics of listening comprehension tests 30
2.1.2. The construct of listening comprehension tests 32
2.2. Channel of presentation 33
2.2.1. Dual coding theory 33
2.2.2. Parallel distributed processing model 35
2.3. Context validity in listening 36
2.3.1. Listening test setting 36
2.3.1. Positive impact of visuals 39
2.3.2. Negative impact of visuals 42
2.4. Cognitive validity in listening 45
2.4.1. Communicative competence models 46
2.4.2. Listening comprehension process 49
2.4.3. Listening skills and strategies 51
2.4.4. Cognitive and Metacognitive processes in listening 53
2.5. Eye-tracking in second language research 56
2.6. Research Hypotheses 57
2.7. Chapter summary 59
Chapter 3. Research Design and Methodology 61
3.1. Philosophical background 61
3.2. Overview of the research design 62
3.2.1. Participants 62
3.2.2. Instruments 63
3.3. Data Collection 70
3.3.1. Listening comprehension test 70
3.3.2. Eye-tracking 71
3.3.3. Stimulated recall interview 77
3.3.4. Cognitive and Metacognitive strategy questionnaire 80
3.4. Data Analysis 80
3.4.1. Analysis of the test scores 81
3.4.2. Analysis of the eye-movements 85
3.4.3. Analysis of the questionnaire 90
3.4.4. Analysis of the stimulated-recall interview data 91
3.5. Pilot study 93
3.6. Ethics 96
Chapter 4. Effect of visual cues on listening test performance 97
4.1. Effect of visual cues on listening test score 97
4.1.1. Baseline test results 97
4.1.2. Listening test results: audio-only vs. video 98
4.2. Random effects of item and candidate characteristics 100
4.3. Effect of proficiency and prompt types 104
4.3.1. Effect of visual cues on listening performance by initial listening proficiency 104
4.3.2. Effect of visual cues on listening performance by prompt types 106
4.4. Chapter summary 109
Chapter 5. Effect of visual cues on candidates' viewing behaviours 112
5.1. Difference in eye-movements between the two subgroups 113
5.1.1. Fixation Count 114
5.1.2. Fixation Duration 115
5.1.3. Total fixation duration 116
5.1.4. Proportion of total fixation duration 118
5.1.5. Visit Count 123
5.1.6. Visit Duration 124
5.1.7. Total Visit Duration 125
5.1.8. Proportion of Total Visit Duration 127
5.2. Difference in eye-movements between the prompt types (lecture vs. dialogue) 131
5.2.1. Fixation Count 131
5.2.2. Fixation Duration 133
5.2.3. Total Fixation Duration 134
5.2.4. Proportion of Total Fixation Duration 136
5.2.5. Visit Count 137
5.2.6. Visit Duration 139
5.2.7. Total Visit Duration 140
5.2.8. Proportion of Total Visit Duration 142
5.2.9. Findings of the audio-only group's eye-movements 144
5.3. Difference in the eye-movements among proficiency levels 145
5.3.1. Fixation Count 145
5.3.2. Fixation Duration 147
5.3.3. Total fixation duration 148
5.3.4. Proportion of Total fixation duration 149
5.3.5. Visit Count 151
5.3.6. Visit Duration 152
5.3.7. Total Visit Duration 154
5.3.8. Proportion of Total Visit Duration 155
5.3.9. Findings of the audio-only group's eye-movements 157
5.4. Exploring relationships between the eye-movements and the listening test performance 158
5.4.1. Fixation Count 158
5.4.2. Fixation Duration 160
5.4.3. Total fixation duration 163
5.4.4. Proportion of total fixation duration 164
5.4.5. Visit Count 167
5.4.6. Visit Duration 169
5.4.7. Total Visit Duration 172
5.4.8. Proportion of total visit duration 174
5.4.9. Summary of Findings 177
5.5. Viewing Pattern Analysis 178
5.5.1. Differences in viewing pattern between the subgroups 179
5.5.2. Differences in viewing pattern by prompt types 180
5.5.3. Differences in viewing pattern among the proficiency levels 183
5.5.4. Effects of viewing patterns on overall test score 185
5.5.5. Random effects of the item and candidate characteristics. 187
5.6. Chapter Summary 192
Chapter 6. Effect of visual cues on candidates' test-taking processes and strategies 197
6.1. Findings of the cognitive and metacognitive process questionnaire 197
6.1.1. Factor Analysis 201
6.2. Findings of the stimulated-recall interview 207
6.2.1. Participants 208
6.2.2. Themes and sub-themes 209
6.2.3. Theme 1: Test-taking process 211
6.2.4. Theme 2: Effect of Video 221
6.3. Chapter Summary 233
Chapter 7. Discussion 238
7.1. Discussion of findings on Research Question 1 238
7.1.1. Summary of findings 239
7.1.2. Implications 242
7.2. Discussion of findings on Research Question 2 246
7.2.1. Summary of findings 247
7.2.2. Implications 257
7.3. Pedagogical and Practical Implications 264
7.4. Discussions on the research methodology 266
7.4.1. Eye-tracking method 266
7.4.2. Triangulation method 269
7.5. Validity arguments 271
7.5.1. Interpretative arguments on video-mediated listening comprehension test 272
7.6. Authenticity in Listening Comprehension Tests 274
Chapter 8. Conclusion 278
8.1. Theoretical contribution 279
8.2. Practical contribution 282
8.2.1. Videos in Language assessment 282
8.2.2. Videos in language classrooms 286
8.3. Limitation 286
8.4. Major contributions of the study 290
8.5. Suggestion for future studies 291
References 293
Appendix 302
Appendix 1. Test specification 302
Appendix 2. Listening test items (Korean & English translation) 303
Appendix 3a. Cognitive and Metacognitive Strategy Questionnaire (English) 309
Appendix 3b. Cognitive and Metacognitive Strategy Questionnaire (Korean translation) 311
Appendix 4. Semi-scripted Interview Questions 312
Appendix 5. Sample field notes 313
Appendix 6. Pilot study 1 findings 316
Appendix 7a. Consent and Instruction (Korean) 318
Appendix 7b. Consent and Instruction (English translation) 319
Appendix 8. Eye-movement difference by the prompt types (audio-only) 320
Appendix 9. Eye-movement difference by proficiency (audio-only) 326
Table 2.1. Meta-analysis on previous studies 44
Table 3.1. Participant information 62
Table 3.2. Difference between the audio-only and video listening prompts 68
Table 3.3. Cognitive and Metacognitive strategy questionnaire 69
Table 3.4. Descriptions of the AOIs 75
Table 3.5. Eye movement measures 75
Table 3.6. Descriptions of the three viewing patterns 87
Table 4.1. Descriptive statistics of baseline score 97
Table 4.2. Descriptive statistics of total listening test scores 98
Table 4.3. Proportion of correct answers for individual items 99
Table 4.4. Result of logistic regression - individual item 99
Table 4.5. Comparison between cross-classified models and single level model 101
Table 4.6. Comparison between cross-classified models and simpler two-level models 101
Table 4.7. Descriptive statistics of baseline proficiency (NELT scores) 105
Table 4.8. Main and Interaction models 106
Table 4.9. Descriptive Statistics of proportion scores for lecture and dialogue items 107
Table 4.10. Descriptive statistics: audio-only group 107
Table 4.11. Descriptive statistics - video group 108
Table 4.12. Cross-classified logistic models: lecture and dialogue 108
Table 4.13. Main and interaction effect models - combined 109
Table 5.1. Descriptive statistics of fixation count 114
Table 5.2. Independent samples t-test - fixation count 115
Table 5.3. Descriptive statistics of fixation duration 115
Table 5.4. Differences in Total fixation duration 117
Table 5.5. Independent samples t-test for total fixation duration 118
Table 5.6. Differences in Proportion of total fixation duration 118
Table 5.7. Independent samples t-test for proportion of total fixation duration 119
Table 5.8. Proportion of total fixation duration - Dialogue 120
Table 5.9. Proportion of total fixation duration - Lecture 121
Table 5.10. Descriptive statistics of the proportion of total fixation duration - common AOIs only 121
Table 5.11. Independent samples t-test for proportion of total fixation duration 122
Table 5.12. Descriptive statistics of visit count 123
Table 5.13. Independent samples t-test for visit count 124
Table 5.14. Descriptive statistics of visit duration 124
Table 5.15. Descriptive statistics of total visit duration 126
Table 5.16. Independent samples t-tests for total visit duration 126
Table 5.17. Descriptive statistics of proportion of total visit duration 127
Table 5.18. Independent samples t-test results - audio-only vs. video 128
Table 5.19. Proportion of total visit duration - Dialogue 128
Table 5.20. Proportion of total visit duration - Lecture 128
Table 5.21. Descriptive statistics of the proportion of total visit duration - common AOIs only 129
Table 5.22. Independent samples t-test for proportion of total visit duration - common AOIs only 130
Table 5.23. Descriptive statistics of fixation count by prompt type 131
Table 5.24. Descriptive statistics of fixation duration by prompt type 133
Table 5.25. Paired t-test results - fixation duration 134
Table 5.26. Descriptive statistics of total fixation duration by prompt types 135
Table 5.27. Descriptive statistics of proportion of total fixation duration on video 136
Table 5.28. Paired t-test on video AOI between dialogue and lecture 137
Table 5.29. Descriptive statistics of visit count between the two prompt types 137
Table 5.30. Paired t-test results on the visit count data between the prompt types 138
Table 5.31. Visit count on PPT slides 139
Table 5.32. Descriptive statistics of visit duration by prompt type 139
Table 5.33. Descriptive statistics of total visit duration by prompt type 141
Table 5.34. Paired t-test results on the total visit duration data 141
Table 5.35. Descriptive statistics of proportion of total visit duration by prompt types 142
Table 5.36. Paired t-test results of the video group - proportion of total visit duration 143
Table 5.37. Summary of findings on eye-movement difference between the prompt types 144
Table 5.38. Descriptive statistics of fixation count by proficiency level 146
Table 5.39. Descriptive statistics of fixation duration for three proficiency groups 147
Table 5.40. Descriptive statistics of total fixation duration by proficiency levels 148
Table 5.41. Descriptive statistics of proportion of total fixation duration by proficiency levels 150
Table 5.42. Descriptive statistics of visit count by proficiency level 151
Table 5.43. Descriptive statistics of visit duration by proficiency level 153
Table 5.44. Descriptive statistics of total visit duration by proficiency level 154
Table 5.45. Descriptive statistics of proportion of total visit duration by proficiency level 156
Table 5.46. Summary of findings 158
Table 5.47. Linear regression models for fixation count 159
Table 5.48. Interaction effect between Group and AOIs - Fixation Count 160
Table 5.49. Linear regression models for fixation duration 161
Table 5.50. Interaction effect between Group and AOIs - Fixation Duration 162
Table 5.51. Linear regression model for total fixation duration 163
Table 5.52. Linear regression model for total fixation duration - both groups 164
Table 5.53. Linear models for proportion of total fixation duration (audio-only group) 165
Table 5.54. Linear models for proportion of total fixation duration (video group) 166
Table 5.55. Combined model of entire sample - Proportion of total fixation duration 167
Table 5.56. Linear regression models for visit count 167
Table 5.57. Interaction effect between Group and AOIs - Visit count 169
Table 5.58. Linear regression models for visit duration 170
Table 5.59. Interaction effect between Group and AOIs - Visit Duration 171
Table 5.60. Linear regression models for total visit duration 173
Table 5.61. Interaction effect between Group and AOIs - Total Visit Duration 174
Table 5.62. Linear models for proportion of total visit duration (audio-only group) 175
Table 5.63. Linear models for proportion of total visit duration (video group) 176
Table 5.64. Combined model of entire sample - Proportion of total visit duration 177
Table 5.65. Descriptive statistics of viewing pattern - all candidates 179
Table 5.66. Descriptive statistics of frequency of showing each viewing pattern by subgroups 179
Table 5.67. Descriptive Statistics of mean frequency of viewing patterns - all candidates 181
Table 5.68. Descriptive Statistics of viewing patterns by prompt type - audio-only 182
Table 5.69. Descriptive Statistics of viewing patterns by prompt type - video 182
Table 5.70. Descriptive statistics of viewing patterns by proficiency levels 183
Table 5.71. Descriptive Statistics of viewing patterns by proficiency - audio-only 184
Table 5.72. Descriptive Statistics of viewing patterns by proficiency - video 185
Table 5.73. Frequency of representative viewing patterns 186
Table 5.74. Linear models for the effect of viewing pattern on test performance 186
Table 5.75. Single level and cross-classified models 188
Table 5.76. simple two-level models and cross classified model 189
Table 5.77. Overview of Chapter 5 Findings 195
Table 6.1. Descriptive statistics of questionnaire responses 198
Table 6.2. Total variances explained by the 3 factors with loadings 203
Table 6.3. Factor loading Matrix 203
Table 6.4. Items allocated to each factor 204
Table 6.5. Regression Models: Cognitive and Metacognitive factors on test score 206
Table 6.6. Participant profiles for stimulated recall interview 208
Table 6.7. Inter-coder reliability on each theme 209
Table 6.8. Number of quotes and candidates 211
Table 6.9. Descriptions of the animation effect and proportion of total fixation duration values 227
Table 7.1. Hypothesis (a) & (b) and test results 245
Table 7.2. Hypotheses (c), (d) & (e) and test results 264
Figure 2.1. Dual channel assumptions 34
Figure 2.2. Celce-Murcia et al (1995)'s communicative competence model 47
Figure 2.3. Expanded CLA model for visual supported listening comprehension 48
Figure 2.4. Information processing model 49
Figure 2.5. Top-down and bottom-up cognitive process in listening comprehension 50
Figure 2.6. Model of listening and response stages 54
Figure 2.7. Hypotheses and Research Questions 59
Figure 3.1. Sample view of the Listening Test materials (video vs. audio-only) 67
Figure 3.2. Data collection procedure 70
Figure 3.3. Nine-point calibration procedure 73
Figure 3.4. AOIs in the video-mediated test (top: dialogue, bottom: lecture) 74
Figure 3.5. Sample view of eye-movement measures - (Proportion) Total fixation duration 76
Figure 3.6. Sample view of eye-movement measures - visit count, fixation count, fixation duration 76
Figure 3.7. Sample view of eye-movement measures - visit duration, total visit duration 77
Figure 3.8. Field notes and stimulated-recall interview at the field 79
Figure 3.9. Data analysis overview 81
Figure 3.10. Cross-classified multilevel model 82
Figure 3.11. Sample view of Thorough Scanning 87
Figure 3.12. Sample view of Selective Focus 88
Figure 3.13. Sample view of Scatter 88
Figure 3.14. Pilot study procedure 94
Figure 3.15. Total mean score distribution plot 95
Figure 4.1. Q-Q plot (left) and caterpillar plot (right) for item effect 103
Figure 4.2. Q-Q plot (left) and caterpillar plot (right) for candidate effect 104
Figure 5.1. Fixation count - audio-only vs. video 114
Figure 5.2. Fixation duration - audio-only vs. video 116
Figure 5.3. Total fixation duration - audio-only vs. video 117
Figure 5.4. Proportion of total fixation duration - audio-only vs. video 119
Figure 5.5. Pie graphs of proportion of total fixation duration (video group) 120
Figure 5.6. Bar graphs of Proportion of total fixation duration 122
Figure 5.7. Visit Count - audio-only vs. video 123
Figure 5.8. Visit duration - audio-only vs. video (median) 125
Figure 5.9. Total visit duration - audio-only vs. video 126
Figure 5.10. Bar graph of proportion of total visit duration on three AOIs 127
Figure 5.11. Pie graphs of proportion of total visit duration - dialogue (left) and lecture (right) 129
Figure 5.12. Proportion of total visit duration - with video AOIs (top), common AOIs only (bottom) 130
Figure 5.13. Fixation count - dialogue vs. lecture 132
Figure 5.14. Fixation Duration - dialogue vs. lecture 133
Figure 5.15. Total fixation duration - dialogue vs. lecture 135
Figure 5.16. Proportion of total fixation duration - dialogue vs. lecture 136
Figure 5.17. Visit count - dialogue vs. lecture 138
Figure 5.18. Visit duration - dialogue vs. lecture 140
Figure 5.19. Total visit duration - dialogue vs. lecture 141
Figure 5.20. proportion of total visit duration - dialogue vs. lecture 143
Figure 5.21. Fixation count - proficiency levels 146
Figure 5.22. Fixation Duration - proficiency level 147
Figure 5.23. Total fixation duration - proficiency level 149
Figure 5.24. Proportion of total fixation duration - proficiency level 150
Figure 5.25. Visit count - proficiency level 152
Figure 5.26. Visit duration - proficiency level 153
Figure 5.27. Total visit duration - proficiency level 154
Figure 5.28. Proportion of total visit duration - proficiency level 156
Figure 5.29. Prediction of total test score gap between the two groups caused by the interaction effect 172
Figure 5.30. Q-Q plot (left) Caterpillar plot (right) for Item random effect 191
Figure 5.31. Q-Q plot (left) and Caterpillar plot (right) for Item random effect 191
Figure 6.1. Scree plot for 25 items 202
Figure 6.2. Main themes and sub-themes 210
Figure 6.3. A sample gaze plot of participant Fred 220
Figure 6.4. An example of the animation effect of the PPT slide 226
Figure 7.1. Sample view of scatter pattern (left: audio-only, right: video) 253
Figure 7.2. Interpretive arguments 272