Handbook of Statistics Vol 26 - Psychometrics

Table of contents
Preface v
Table of contents vii
Contributors xvii
Ch. 1. A History and Overview of Psychometrics 1
Lyle V. Jones and David Thissen
1. Introduction 1
2. The origins of psychometrics (circa 1800?960) 1
3. Psychological scaling 8
4. Psychological measurement (test theory) 10
5. Factor analysis 15
6. Psychological statistics 18
7. Conclusion 21
References 22
Ch. 2. Selected Topics in Classical Test Theory 29
Charles Lewis
1. A modern introduction 31
2. Restriction of range 36
3. Nonlinear classical test theory 39
References 42
Ch. 3. Validity: Foundational Issues and Statistical Methodology 45
Bruno D. Zumbo
1. Introductory remarks 45
2. Foundational issues 47
3. Statistical methods 65
4. Closing remarks 71
References 76
vii
viii Table of contents
Ch. 4. Reliability Coefficients and Generalizability Theory 81
Noreen M. Webb, Richard J. Shavelson and Edward H. Haertel
1. Introduction 81
2. Reliability Coefficients in Classical Test Theory 81
3. Generalizability theory 93
4. Concluding remarks 120
References 121
Ch. 5. Differential Item Functioning and Item Bias 125
Randall D. Penfield and Gregory Camilli
1. Introduction 125
2. General definition of DIF 127
3. Item response theory approaches for dichotomous items 128
4. Proportion-difference approaches for dichotomous items 132
5. Common odds ratio approaches for dichotomous items 135
6. Logistic regression approaches for dichotomous items 139
7. Classification schemes for dichotomous items 141
8. Item response theory approaches for polytomous items 143
9. Mean-difference approaches for polytomous items 146
10. Multivariate hypergeometric distribution approaches for polytomous items 147
11. Common odds ratio approaches for polytomous items 149
12. Logistic regression approaches for polytomous items 150
13. Differential test functioning and DIF effect variance 152
14. Explaining the sources of DIF 155
15. Steps to conducting DIF analyses: An applied example 157
16. Cautions and limitations 159
References 163
Ch. 6. Equating Test Scores 169
Paul W. Holland, Neil J. Dorans and Nancy S. Petersen
1. Introduction and definitions 169
2. Data collection designs used in test score equating 174
3. Procedures for equating scores 183
4. Best practices and challenges to best practices 197
References 201
Ch. 7. Electronic Essay Grading 205
Shelby J. Haberman
1. Regression analysis 206
2. Composite scales 215
3. Content analysis 218
4. Analysis of discrete responses 226
5. Bayesian analysis 230
6. Conclusions 231
References 232
Table of contents ix
Ch. 8. Some Matrix Results Useful in Psychometric Research 235
C. Radhakrishna Rao
1. Notation and Basic Matrix Results 235
2. Decomposition of matrices 239
3. Matrix approximations 243
4. Procrustean transformation 247
5. Correspondence analysis 250
6. Metric and multidimensional scaling 252
Acknowledgements 254
References 254
Ch. 9. Factor Analysis 257
Haruo Yanai and Masanori Ichikawa
1. Introduction 257
2. Historical development of factor analysis 257
3. The factor analysis model 261
4. Statistical inference in factor analysis 271
5. Factor rotation and estimation of factor scores 281
Acknowledgements 293
References 293
Ch. 10. Structural Equation Modeling 297
Ke-Hai Yuan and Peter M. Bentler
1. Introduction 297
2. Model identification 303
3. Estimation and evaluation 305
4. Missing data 327
5. Multiple groups 334
6. Multilevel models with hierarchical data 338
7. Examples 344
References 348
Ch. 11. Applications of Multidimensional Scaling in Psychometrics 359
Yoshio Takane
1. Introduction 359
2. MDS with the simple Euclidean model 361
3. Individual differences MDS 375
4. Unfolding analysis 385
5. Concluding remarks 395
References 397
x Table of contents
Ch. 12. Multilevel Models in Psychometrics 401
Fiona Steele and Harvey Goldstein
1. Introduction 401
2. Basic Models for Two-level Hierarchical Data Structures 402
3. Models for repeated measures 406
4. Models for multivariate response data 409
5. Models for non-hierarchical structures 412
6. Further extensions 416
7. Estimation procedures and software 417
8. Resources 418
References 418
Ch. 13. Latent Class Analysis in Psychometrics 421
C. Mitchell Dayton and George B. Macready
1. Introduction 421
2. The model for LCA 422
3. Estimation and model fit 423
4. Analyses with score data 428
5. Unconstrained latent class models 432
6. Scaling models 433
7. Models incorporating grouping of respondents 436
8. Covariate latent class models 438
9. Other models 441
References 444
Ch. 14. Random-Effects Models for Preference Data 447
Ulf B?ckenholt and Rung-Ching Tsai
1. Introduction 447
2. Thurstonian random utility models 449
3. Identifiability 455
4. Identifying the scale origin 458
5. Estimation 459
6. Applications 460
7. Concluding remarks 466
References 466
Ch. 15. Item Response Theory in a General Framework 469
R. Darrell Bock and Irini Moustaki
1. Introduction 469
2. The general IRT framework 470
3. Item response models 472
4. Estimation of item and group parameters 490
5. Estimation of respondent scores 499
6. Item factor analysis 505
7. Response relations with external variables 508
References 509
Table of contents xi
Ch. 16. Rasch Models 515
Gerhard H. Fischer
1. Some history of the Rasch model 515
2. Some basic concepts and properties of the RM 517
3. Characterizations and scale properties of the RM 522
4. Item parameter estimation 532
5. Person parameter estimation 546
6. Testing of fit 549
7. The linear logistic test model 564
8. Longitudinal linear logistic models 571
9. Some remarks on applications and extensions of the RM 576
References 578
Ch. 17. Hierarchical Item Response Theory Models 587
Matthew S. Johnson, Sandip Sinharay and Eric T. Bradlow
1. Introduction 587
2. Developing the hierarchical IRT model 589
3. Estimation 592
4. Examples 595
5. Conclusions 604
References 604
Ch. 18. Multidimensional Item Response Theory 607
Mark D. Reckase
1. Introduction 607
2. General forms of MIRT models 609
3. Common forms of MIRT models 612
4. Descriptions of item characteristics 619
5. Descriptions of test characteristics 625
6. The estimation of model parameters 629
7. Applications 631
8. Discussion and conclusions 640
References 641
Ch. 19. Mixture Distribution Item Response Models 643
Matthias von Davier and J?rgen Rost
1. Introduction 643
2. Different perspectives on mixture IRT 644
3. Applications of mixture IRT models 655
4. Mixture IRT for large-scale survey assessments 656
5. Conclusion 658
References 659
xii Table of contents
Ch. 20. Scoring Open Ended Questions 663
Gunter Maris and Timo Bechger
1. Introduction 663
2. Motivating example 663
3. Theory 665
4. A measurement model for scoring open ended questions 673
5. Motivating example revisited 677
6. Discussion 679
References 680
Ch. 21. Assessing the Fit of Item Response Theory Models 683
Hariharan Swaminathan, Ronald K. Hambleton and H. Jane Rogers
1. Introduction: Models and assumptions 683
2. Checking the assumption of unidimensionality 686
3. Assessing model data fit: Checking model predictions 691
4. Empirical example 707
5. Conclusions 714
Acknowledgements 715
References 715
Ch. 22. Nonparametric Item Response Theory and Special Topics 719
Klaas Sijtsma and Rob R. Meijer
1. Place of nonparametric models in item response theory 719
2. Analyzing test data using nonparametric IRT 727
3. Special topics in NIRT 739
References 743
Ch. 23. Automatic Item Generation and Cognitive Psychology 747
Susan Embretson and Xiangdong Yang
1. Introduction 747
2. Item development and automatic item generation 748
3. Item response models for automatic item generation 753
4. Calibration of the cognitive IRT models 756
5. An application: Cognitive models for algorithmically generated spatial ability items 761
6. Overall discussion 765
References 766
Table of contents xiii
Ch. 24. Statistical Inference for Causal Effects, with Emphasis on Applications in
Psychometrics and Education 769
Donald B. Rubin
1. Causal inference primitives 769
2. The assignment mechanism 776
3. Randomization-based modes of causal inference 781
4. Posterior predictive causal inference 786
5. Complications 794
References 796
Ch. 25. Statistical Aspects of Adaptive Testing 801
Wim J. van der Linden and Cees A.W. Glas
1. Introduction 801
2. Response models 802
3. Item calibration 804
4. Ability estimation 811
5. Empirical examples 812
6. Rules for adaptive item selection 814
7. Other statistical issues in adaptive testing 822
8. Concluding comment 833
References 835
Ch. 26. Bayesian Psychometric Modeling From An Evidence-Centered Design
Perspective 839
Robert J. Mislevy and Roy Levy
1. Introduction and overview 839
2. An evidentiary perspective 840
3. Evidence-centered design 847
4. Bayesian psychometric modeling 856
5. Discussion 861
6. Concluding remarks 863
References 863
Ch. 27. Value-Added Modeling 867
Henry Braun and Howard Wainer
1. Introduction 867
2. General issues 869
3. Models for estimating teacher effects 875
4. Example 880
5. Current research 884
6. Summing up 888
References 889
xiv Table of contents
Ch. 28. Three Statistical Paradoxes in the Interpretation of Group Differences:
Illustrated with Medical School Admission and Licensing Data 893
Howard Wainer and Lisa M. Brown
1. Introduction 893
2. The data 894
3. Simpson?s Paradox 895
4. Kelley?s Paradox 901
5. Lord?s Paradox 910
6. Conclusion 916
Acknowledgements 917
References 918
Ch. 29. Meta-Analysis 919
Larry V. Hedges
1. Introduction 919
2. Effect sizes 920
3. The problem of dependent estimates 923
4. Fixed effects analyses 924
5. Mixed models 942
6. Conclusions 952
References 953
Ch. 30. Vertical Scaling: Statistical Models for Measuring Growth and
Achievement 955
Richard J. Patz and Lihua Yao
1. Introduction 955
2. Vertical scaling of achievement tests: Recent and past statistical practice 956
3. Statistical modeling approaches for vertical scaling 960
4. Exploring grade-to-grade growth in achievement: An application of hierarchical modeling 964
5. Discussion 972
References 973
Ch. 31. COGNITIVE DIAGNOSIS
Ch. 31A. Review of Cognitively Diagnostic Assessment and a Summary of
Psychometric Models 979
Louis V. DiBello, Louis A. Roussos and William Stout
1. Preliminaries 979
2. Implementation framework for diagnostic assessment 980
3. Reliability, validity, and granularity 993
4. Summary of cognitively diagnostic psychometric models 995
5. Summary and conclusions: Issues and future directions 1023
References 1027
Table of contents xv
Ch. 31B. Some Notes on Models for Cognitively Based Skills Diagnosis 1031
Shelby J. Haberman and Matthias von Davier
Introduction 1031
1. Retrofitting a test with a hypothesized skill structure 1033
2. Dimensionality 1034
3. Utility and use of skills classifications 1035
4. Latent structure models, latent responses, conjunctive versus compensatory skills 1036
5. Outlook 1037
References 1038
Ch. 32. The Statistical Procedures Used in National Assessment of Educational
Progress: Recent Developments and Future Directions 1039
Matthias von Davier, Sandip Sinharay, Andreas Oranje and Albert
Beaton
1. Introduction 1039
2. The current NAEP model and estimation method 1041
3. Example: NAEP data and results 1047
4. Alternatives approaches 1049
5. Conclusions 1052
Acknowledgements 1053
Appendix: Sampling students in NAEP 1053
References 1054
Ch. 33. Statistical Procedures Used in College Admissions Testing 1057
Jinghua Liu, Deborah J. Harris and Amy Schmidt
1. Introduction 1057
2. Test design and assembly 1059
3. Item pretesting 1064
4. Item analysis 1066
5. Scaling and equating 1074
6. Reliability and validity 1077
7. Other issues 1086
8. Summary 1088
References 1089
Ch. 34. FUTURE CHALLENGES IN PSYCHOMETRICS
Ch. 34A. Integration of Models 1095
Robert L. Brennan
What constitutes a replication? 1095
What are true scores? 1096
What is error? 1097
No ?right?model 1097
References 1097
xvi Table of contents
Ch. 34B. Linking Scores Across Computer and Paper-Based Modes of Test
Administration 1099
Daniel R. Eignor
References 1102
Ch. 34C. Linking Cognitively-Based Models and Psychometric Methods 1103
Mark J. Gierl and Jacqueline P. Leighton
Introduction 1103
Developing cognitive models of task performance 1104
Incorporating cognitive models into psychometric methods 1105
Conclusion 1106
References 1106
Ch. 34D. Technical Considerations in Equating Complex Assessments 1107
Ida Lawrence
References 1109
Ch. 34E. Future Challenges to Psychometrics: Validity, Validity, Validity 1111
Neal Kingston
References 1112
Ch. 34F. Testing with and without Computers 1113
Piet Sanders
Ch. 34G. Practical Challenges to Psychometrics Driven by Increased Visibility of
Assessment 1117
Cynthia Board Schmeiser
Growth models 1117
Diagnosis, prescription, and instruction 1118
Dual-platform testing 1118
Index 1121
Handbook of Statistics Contents of Previous Volumes 1143