Title

Contents

Foreword 3

List of abbreviations and acronyms 15

Chapter 1. Introduction and Summary 17

Abstract 18

Overview of TALIS 18

Management of the study 18

Standardised procedures to ensure high-quality data 19

National centres and National Project Managers 19

Main tasks of the National Project Managers 19

Phases of the TALIS study 20

Pilot study 20

Field trial 20

Main survey 21

Chapter 2. Overview of TALIS 2008 and Framework Development 23

Abstract 24

The rationale for and aims of TALIS 24

The TALIS survey programme design 25

TALIS linked to PISA 25

The conceptual and analytic framework of the programme 25

Choosing the policy focus for TALIS 26

References 28

Chapter 3. Development of Teacher and Principal Questionnaires 29

Abstract 30

The role and membership of the OECD TALIS Instrument Development Expert Group 30

Timeline and phases of questionnaire development 31

Frameworks guiding questionnaire development 31

Teacher background characteristics and professional development 33

Principal and school background characteristics 34

School leadership and management 34

Appraisal of and feedback to teachers 35

Teaching practices, attitudes and beliefs 37

Pilot test 39

Field trial 39

References 42

Chapter 4. Translation and Cultural Adaptation 45

Abstract 46

Overview 46

Instruments to be translated 46

Cultural and national adaptations 47

Translation procedures 47

Identifying the target language 48

Engaging translators 49

Producing translations 49

Submitting materials for external verification 50

International translation verification 51

Processes of translation verification 51

Translation verification report 51

Translation verification summary 51

Feedback from National Project Managers on translation and translation verification 52

International Quality Control Monitor review of translation 52

Layout verification: paper and on line data collection 52

References 54

Chapter 5. Sample Design 55

Abstract 56

Overview 56

International sampling plan 56

Target population and survey population: International requirements and national implementations 56

Sample size requirements 58

National sampling strategies 58

Sampling frames 59

Stratification 59

Sample selection 59

Sampling for the field trial 60

National sampling plans 60

References 64

Chapter 6. Survey Operations Procedures 65

Abstract 66

Manuals and software 66

Contacting schools and within-school sampling procedures 67

Identification numbers, Teacher Listing Forms and Teacher Tracking Forms 67

Assigning materials to teachers and school principals 68

Administering the questionnaires and national quality control 69

Monitoring the on line questionnaires 69

Material receipt and preparing for data entry 69

Survey Activities Questionnaire 69

References 71

Chapter 7. On line Data Collection 73

Abstract 74

Overview 74

Design and mixed-mode considerations 74

Technical implementation 76

Operations 78

Field trial 79

Main study participation, mode distribution and evaluation 80

References 88

Chapter 8. Quality Asurance 89

Abstract 90

Overview 90

Quality control in TALIS survey administration 90

International quality control monitoring programme 91

Survey anonymity 91

School co-ordinator interviews 92

Initial preparations 92

Survey administration activities 93

General observations 93

Survey Activities Questionnaire 94

Contacting schools 94

Preparing survey materials 95

Conversion to on line data collection 96

Survey administration 96

Data entry and verification 97

National quality control monitoring programme 97

References 101

Chapter 9. Creating and Checking the International Database 103

Abstract 104

Overview 104

Data entry and verification at national centres 104

Data checking, editing and quality control at the IEA Data Processing and Researcher Center 106

Import, documentation and structure check 107

Identification variable and linkage cleaning 108

Resolving inconsistencies in questionnaire data 108

Handling missing data 110

Interim data products 111

Building the international database 111

References 113

Chapter 10. Estimation Weights, Participation Rates and Sampling Error 115

Abstract 116

Overview 116

Elements of the estimation weight (final weight) 116

School base weight (school design weight) 117

School non-response adjustment factor 117

Final school weight 118

Teacher base weight (teacher design weight) 118

Teacher non-response adjustment factor 119

Teacher adjustment factor for incidental exclusions 119

Teacher multiplicity adjustment factor 119

Final teacher weight 120

Participation rates 120

Participation rates for schools 121

Participation rate for teachers 121

Overall participation rates 122

Reporting participation rates 122

Meeting participation rates standard for TALIS 122

Sampling error with Balanced Repeated Replication 123

Creating replicates for Balanced Repeated Replication 123

Estimating the sampling error 125

Design effect and effective sample size 125

References 129

Chapter 11. Construction and validation of scales and indices 131

Abstract 132

Overview 132

Simple questionnaire indices, ratios and indicators 132

Student-teacher ratio 132

Validation of complex questionnaire scale indices 135

Indices derived through Principal Components Analysis 135

Description of complex scale indices and their parameters 145

School leadership indices 145

References 205

Annex A. TALIS consortium, experts and consultants 207

Annex B. Characteristics of National Samples 211

Australia 212

Austria 213

Belgium (Flanders) 214

Brazil 215

Bulgaria 216

Denmark 217

Estonia 218

Hungary 219

Iceland 220

Ireland 221

Italy 222

Korea 223

Lithuania 224

Malaysia 225

Malta 226

Mexico 227

Norway 228

Poland 229

Portugal 230

Slovak Republic 231

Slovenia 232

Spain (Excluding La Rioja, Canarias) 233

The Netherlands 234

Turkey 235

Annex C. Sampling Forms 237

Sampling Form 1 - Participation 238

Sampling Form 2 - National Target Population 239

Sampling Form 3 - Stratification 240

Sampling Form 4 - Sampling Frame Description 241

Sampling Form 5 - Excluded schools 242

Sampling Form 6 - Population counts by strata 243

Sampling Form 7 - Sample allocation 244

Annex D. TALIS Principal and Teacher Questionnaire 245

Principal Questionnaire 246

Background information 247

School background information 248

School management 250

Teacher appraisal 253

School resources 255

Teacher Questionnaire 259

Background information 260

Professional development 262

Teacher appraisal and feedback 264

Teaching practices, beliefs and attitudes 267

Your teaching in a particular ＜class＞ at this school 271

Table 5.1 Reasons for and magnitude of school exclusion, by country 61

Table 5.2 Derivation of the required sample size 62

Table 5.3 Illustration of systematic random sampling with PPS 62

Table 5.4 Overview of the national sampling plans 63

Table 7.1 Extent to which paper and on line administration modes were used for the Principal Questionnaire during the main study (sorted in descending order of on line usage and country name) 84

Table 7.2 Extent to which paper and on line administration modes were used for the Teacher Questionnaire during the main study (sorted in descending order of on line usage and country name) 85

Table 7.3 Percentage of questionnaires administered on line, by respondent type and gender 85

Table 7.4 Percentage of questionnaires administered on line, by teachers' age group 86

Table 7.5 Percentage of schools that administered questionnaires on line, by country 86

Table 7.6 Error typology: type, initial percentage of all analysed login attempts, percentage of successful corrections, and percentage of residual failed login attempts(sorted in descending order of magnitude of initial error rate) 86

Table 7.7 Maximum, mean, and median duration for completing an on line questionnaire, as well as percentage of completed questionnaires within one login, one day, and one week, by country 87

Table 8.1 Preliminary activities of the School Co-ordinator 99

Table 8.2 Questionnaire distribution and returns 99

Table 8.3 Use of forms in survey administration 99

Table 8.4 Use of specialist translators and reviewers 100

Table 10.1 Quality ratings and unweighted participation rates 126

Table 10.2 Unweighted participation rates, by country 126

Table 10.3 Weighted participation rates, by country 126

Table 10.4 Example of BRR-ready sample design and random assignation of pseudo PSUs 127

Table 10.5 Estimated design effects and effective sample sizes for selected key variables, their average and the original and effective sample sizes, by country (Teacher Questionnaire variables) 127

Table 10.6 Estimated design effects and effective sample size for selected key variables, their average and the original and effective sample sizes, by country (Principal Questionnaire variables) 128

Table 11.1 Item wording for school autonomy indices 152

Table 11.2 Selection of indices - rotated component matrix 152

Table 11.3 Factor loadings used for computation of factor scores for the indices of school autonomy 152

Table 11.4 Reliabilities for indices of school autonomy 153

Table 11.5 Item wording for school resource indices 153

Table 11.6 Selection of indices-rotated component matrix 154

Table 11.7 Factor loadings used for computation of factor scores for the indices of school resources 154

Table 11.8 Reliabilities for indices of school resources 154

Table 11.9 International means and standard deviations of school-and teacher-level factor score estimates using equally weighted pooled data 155

Table 11.10 Item wording of school leadership indices and dimensions 155

Table 11.11 Single items measuring school leadership and management behaviours 156

Table 11.12 Reliabilities for indices of school leadership 156

Table 11.13 Model fit for the scale Framing and communicating the school's goals and curricular development 157

Table 11.14 Model fit for the scale Promoting instructional improvements and professional development 157

Table 11.15 Model fit for the scale Supervision of instruction in the school 158

Table 11.16 Model fit for the scale Accountability role of the principal 158

Table 11.17 Model fit for the scale Bureaucratic rule-following 159

Table 11.18 Multiple group CFA examining different levels of cross-cultural invariance of the scale Framing and communicating the school's goals and curricular development 159

Table 11.19 Multiple group CFA examining different levels of cross-cultural invariance of the scale Promoting instructional improvements and professional development 159

Table 11.20 Multiple group CFA examining different levels of cross-cultural invariance of the scale Supervision of instruction in the school 160

Table 11.21 Multiple group CFA examining different levels of cross-cultural invariance of the scale Accountability role of the principal 160

Table 11.22 Multiple group CFA examining different levels of cross-cultural invariance of principals' Bureaucratic rule-following 160

Table 11.23 Factor loadings used for the computation of factor scores for the indicator Framing and communicating the school's goals and curricular development 160

Table 11.24 Item intercepts used for the computation of factor scores for the indicator Framing and communicating the school's goals and curricular development, by country 161

Table 11.25 Unique variances used for the computation of factor scores for the indicator Framing and communicating the school's goals and curricular development, by country 161

Table 11.26 Factor loadings used for the computation of factor scores for the indicator Promoting instructional improvements and professional development 162

Table 11.27 Item intercepts used for the computation of factor scores for the indicator Promoting instructional improvements and professional development, by country 162

Table 11.28 Unique variances used for the computation of factor scores for the indicator Promoting instructional improvements and professional development, by country 163

Table 11.29 Factor loadings used for the computation of factor scores for the indicator Supervision of instruction in the school 163

Table 11.30 Item intercepts used for the computation of factor scores for the indicator Supervision of instruction in the school, by country 164

Table 11.31 Unique variances used for the computation of factor scores for the indicator Supervision of instruction in the school, by country 164

Table 11.32 Factor loadings used for the computation of factor scores for the indicator Accountability role of the principal 165

Table 11.33 Item intercepts used for the computation of factor scores for the indicator Accountability role of the principal, by country 165

Table 11.34 Unique variances used for the computation of factor scores for the indicator Accountability role of the principal, by country 166

Table 11.35 Factor loadings used for the computation of factor scores for the indicator Bureaucratic rule-following 166

Table 11.36 Item intercepts used for the computation of factor scores for the indicator Bureaucratic rule-following, by country 167

Table 11.37 Unique variances used for the computation of factor scores for the indicator Bureaucratic rule-following, by country 167

Table 11.38 Factor score determinacy for the scales related to school principal's leadership, by country 168

Table 11.39 Item wording of school climate items and dimensions - Principal Questionnaire 168

Table 11.40 Single items measuring aspects of school climate 169

Table 11.41 Item wording of school climate items and dimensions - Teacher Questionnaire 169

Table 11.42 Reliabilities for school climate indices, by country 169

Table 11.43 Model fit and latent correlations for factors influencing school climate, by country 170

Table 11.44 Multiple group CFA examining different levels of cross-cultural invariance of factors influencing school climate 170

Table 11.45 Factor loadings used for the computation of factor scores for the indicator School climate: student delinquency 170

Table 11.46 Item intercepts used for the computation of factor scores for the indicator School climate: student delinquency, by country 171

Table 11.47 Unique variances used for the computation of factor scores for the indicator School climate: student delinquency, by country 171

Table 11.48 Factor loadings used for the computation of factor scores for the indicator School climate: teachers' working morale 172

Table 11.49 Item intercepts used for the computation of factor scores for the indicator School climate: teachers' working morale, by country 172

Table 11.50 Unique variances used for the computation of factor scores for the indicator School climate: teachers' working morale, by country 173

Table 11.51 Factor score determinacy for the indicators School climate: student delinquency and School climate: teachers' working morale, by country 173

Table 11.52 Model fit of Teacher-student relations (TQ), by country 174

Table 11.53 Multiple group CFA examining different levels of cross-cultural invariance of Teacher-student relations 174

Table 11.54 Factor loadings used for the computation of factor scores for the indicator Teacher-student relations 174

Table 11.55 Item intercepts used for the computation of factor scores for the indicator Teacher-student relations, by country 175

Table 11.56 Unique variances used for the computation of factor scores for the indicator Teacher-student relations, by country 175

Table 11.57 Factor score determinacy for the indicators Teacher-student relations, by country 176

Table 11.58 Item wording of Classroom disciplinary climate items and dimensions 176

Table 11.59 Reliabilities for Classroom disciplinary climate, by country 177

Table 11.60 Model fit Classroom disciplinary climate, by country 177

Table 11.61 Multiple group CFA examining different levels of cross-cultural invariance of Classroom disciplinary climate 178

Table 11.62 Factor loadings used for the computation of factor scores for the indicator Classroom disciplinary climate 178

Table 11.63 Item intercepts used for the computation of factor scores for the indicator Classroom disciplinary climate, by country 178

Table 11.64 Unique variances used for the computation of factor scores for the indicator Classroom disciplinary climate, by country 179

Table 11.65 Factor score determinacy for the indicators Classroom disciplinary climate, by country 179

Table 11.66 Item wording of the scale Self-efficacy 180

Table 11.67 Reliabilities for Self-efficacy, by country 180

Table 11.68 Model fit for Self-efficacy, by country 181

Table 11.69 Multiple group CFA examining different levels of cross-cultural invariance of Self-efficacy 181

Table 11.70 Factor loadings used for the computation of factor scores for the indicator Self-efficacy 181

Table 11.71 Item intercepts used for the computation of factor scores for the indicator Self-efficacy, by country 182

Table 11.72 Unique variances used for the computation of factor scores for the indicator Self-efficacy, by country 182

Table 11.73 Factor score determinacy for the indicators Self-efficacy, by country 183

Table 11.74 Item wording of teachers' and principals' beliefs items and dimensions 183

Table 11.75 Single items measuring teachers' and principals' Beliefs about instruction 184

Table 11.76 Reliabilities for indices of teachers’ Beliefs about instruction, by country 184

Table 11.77 Model fit and latent correlations for teachers' Direct transmission beliefs about instruction and Constructivist beliefs about instruction, by country 185

Table 11.78 Multiple group CFA examining different levels of cross-cultural invariance of teachers' Direct transmission beliefs about instruction 185

Table 11.79 Multiple group CFA examining different levels of cross-cultural invariance of teachers' Constructivist beliefs about instruction 186

Table 11.80 Factor loadings used for the computation of factor scores for the indicators Direct transmission beliefs about instruction and Constructivist beliefs about instruction 186

Table 11.81 Item intercepts used for the computation of factor scores for the indicators Direct transmission beliefs about instruction and Constructivist beliefs about instruction, by country 186

Table 11.82 Unique variances used for the computation of factor scores for the indicators Direct transmission beliefs about instruction and Constructivist beliefs about instruction, by country 187

Table 11.83 Factor score determinacy for the indicators Direct transmission beliefs about instruction and Constructivist beliefs about instruction, by country 188

Table 11.84 Reliabilities for indices of principals' Constructivist beliefs about instruction, by country 188

Table 11.85 Model fit for principals' Constructivist beliefs about instruction, by country 189

Table 11.86 Multiple group CFA examining different levels of cross-cultural invariance of principals' Constructivist beliefs about instruction 189

Table 11.87 Factor loadings used for the computation of factor scores for the indicator principals' Constructivist beliefs about instruction 189

Table 11.88 Item intercepts used for the computation of factor scores for the indicator principals' Constructivist beliefs about instruction, by country 190

Table 11.89 Unique variances used for the computation of factor scores for the indicator principals' Constructivist beliefs about instruction, by country 190

Table 11.90 Factor score determinacy for the indicators principals' Constructivist beliefs about instruction, by country 191

Table 11.91 Item wording of classroom teaching practices items and dimensions 191

Table 11.92 Single items measuring classroom teaching practices items and dimensions 191

Table 11.93 Reliabilities for indices of classroom teaching practices, by country 192

Table 11.94 Reliabilities for indices of classroom teaching practices, by subject 192

Table 11.95 Model fit and latent correlations for classroom teaching practices, by country 193

Table 11.96 Model fit and latent correlations for classroom teaching practices, by subject 193

Table 11.97 Multiple group CFA examining different levels of cross-cultural invariance of Classroom teaching practice: structuring 194

Table 11.98 Multiple group CFA examining different levels of cross-cultural invariance of Classroom teaching practice: student-oriented 194

Table 11.99 Multiple group CFA examining different levels of cross-cultural invariance of Classroom teaching practice: enhanced activities 194

Table 11.100 Factor loadings used for the computation of factor scores for the indicator Classroom teaching practice: structuring 194

Table 11.101 Item intercepts used for the computation of factor scores for the indicator Classroom teaching practice: structuring, by country 195

Table 11.102 Unique variances used for the computation of factor scores for the indicator Classroom teaching practice: structuring, by country 195

Table 11.103 Factor loadings used for the computation of factor scores for the indicator Classroom teaching practice: student-oriented 196

Table 11.104 Item intercepts used for the computation of factor scores for the indicator Classroom teaching practice: student-oriented, by country 196

Table 11.105 Unique variances used for the computation of factor scores for the indicator Classroom teaching practice: student-oriented, by country 197

Table 11.106 Factor loadings used for the computation of factor scores for the indicator Classroom teaching practice: enhanced activities 197

Table 11.107 Item intercepts used for the computation of factor scores for the indicator Classroom teaching practice: enhanced activities, by country 198

Table 11.108 Unique variances used for the computation of factor scores for the indicator Classroom teaching practice: enhanced activities, by country 198

Table 11.109 Factor score determinacy for the indicators of Classroom teaching practice, by country 199

Table 11.110 Item wording of Co-operation among staff 199

Table 11.111 Single items measuring Co-operation among staff 199

Table 11.112 Reliabilities for indices of Co-operation among staff, by country 200

Table 11.113 Model fit and latent correlations for indices of Co-operation among staff, by country 200

Table 11.114 Multiple group CFA examining different levels of cross-cultural invariance of Exchange and co-ordination for teaching 201

Table 11.115 Multiple group CFA examining different levels of cross-cultural invariance of Professional collaboration 201

Table 11.116 Factor loadings used for the computation of factor scores for the indicator Exchange and co-ordination for teaching 201

Table 11.117 Item intercepts used for the computation of factor scores for the indicator Exchange and co-ordination for teaching, by country 202

Table 11.118 Unique variances used for the computation of factor scores for the indicator Exchange and co-ordination for teaching, by country 202

Table 11.119 Factor loadings used for the computation of factor scores for the indicator Professional collaboration 203

Table 11.120 Item intercepts used for the computation of factor scores for the indicator Professional collaboration, by country 203

Table 11.121 Unique variances used for the computation of factor scores for the indicator Professional collaboration, by country 204

Table 11.122 Factor score determinacy for the indicators of co-operation among staff, by country 204