Authors: Gregory J. Cizek (Editor), Michael B. Bunch
ISBN-13: 9781412916820, ISBN-10: 1412916828
Format: Hardcover
Publisher: SAGE Publications
Date Published: November 2006
Edition: (Non-applicable)
Michael B. Bunch is Vice-President of Measurement Incorporated, a test development and scoring company serving the statewide assessment community. Dr. Bunch joined MI in 1982 and has built the research and development division of that organization into a team of more than 60 project directors, psychometricians, editors, content specialists, and support staff. Prior to joining MI, Dr. Bunch was a senior professional with NTS Research Corporation, where he provided ESEA Title I evaluation technical assistance to state education agencies. From 1976 to 1978, he was a research psychologist with the American College Testing Program (ACT). Dr. Bunch has authored dozens of professional and technical reports on a variety of topicsmany on standard settingand has presented his work at national conferences, including annual meetings of the American Educational Research Association (AERA), National Council on Measurement in Education (NCME), American Psychological Association (APA), Council of Chief State School Officers (CCSSO), Education Commission of the States (ECS) and Adult Education Research Conference (AERC). He received his Ph.D. in psychological measurement from the University of Georgia.
Book Synopsis
In Standard Setting: A Guide to Establishing and Evaluating Performance Standards on Tests, authors Gregory J. Cizek and Michael B. Bunch provide the only "go-to" reference on how to set standards on tests in education, licensure, and certification. This book is comprehensive in scope, practical in nature, and definitive in terms of cataloguing the essential conceptual and procedural fundamentals of setting performance standards.
Key Features:
Offers a detailed description of procedures: In a practical, straightforward, and virtually jargon-free style, this book begins with important foundational and conceptual information about standard setting followed by step-by-step procedures for all currently used methods, including Angoff, Item Mapping, Analystic Judgment, etc. In addition, a full chapter is devoted to future issues and research in standard setting. Enables readers to design, implement, and evaluate a standard setting study: Using this book, readers will be able to: select an appropriate standard setting method; schedule and plan implementation; identify and train qualified standard-setting participants; conduct the standard setting meeting; calculate one or more cut scores; and evaluate the success of the procedure they have implemented. Allows readers to adapt written materials to their situation: Examples are provided in widely available formats such as Excel, SPSS, .pdf and Word files so readers do not have to purchase specialized software or learn to use unfamiliar packages. The book includes downloadable forms, sample correspondence, checklists, data sets, and analytical software for performing calculations required for various standard setting procedures. Intended Audience:
This is an excellent supplementary textbook for graduate courses such as Applied Measurement in Education; Psychometric Theory; Advanced Educational Measurement; Special Topics in Measurement and Evaluation; Seminar in Educational and Psychological Measurement in the departments of Educational Psychology, Measurement and Evaluation, Psychology, and others.
Table of Contents
Preface xi
Fundamentals of Standard Setting 1
Contemporary Standard Setting: An Enduring Need 5
The Need to Make Decisions 6
The Benefits of Standard Setting 8
General Approaches to Standard Setting 9
Current Contexts for Standard Setting 11
What Is Standard Setting? 13
Kinds of Standards 14
Definitions of Standard Setting 14
Policy Issues and Standard Setting 19
Scoring Models 20
Research on Standard Setting 22
Rounding 23
Classification Errors 25
Item Scoring Criteria and Total-Test Performance Standards 29
Conclusions 33
Common Elements in Setting Performance Standards 35
Purpose 36
Choosing a Standard-Setting Method 41
Performance Level Labels 44
Performance Level Descriptions 46
Key Conceptualizations 48
Selecting and Training Standard-Setting Participants 49
Providing Feedback to Participants 53
Normative Feedback 54
Reality Feedback 55
Impact Feedback 56
Professional Guidelines for Standard Setting 57
Evaluating Standard Setting 59
Conclusions and a Foreword 63
Standard-Setting Methods 65
The Nedelsky Method 69
Procedures for the Nedelsky Method 70
Alternative Procedures and Limitations 71
The Ebel Method 75
Procedures for the Ebel Method 75
Alternative Procedures and Limitations 78
The Angoff Method and Angoff Variations 81
Procedures for the Angoff Method 82
Procedures for Angoff Variations 87
The Extended Angoff Method 87
The Yes/No Method 88
Alternative Procedures and Limitations 92
The Direct Consensus Method 97
Procedures for the Direct Consensus Method 98
Alternative Procedures and Limitations 102
The Contrasting Groups and Borderline Group Methods 105
Procedures for the Contrasting Groups Method 106
An Example Using the Contrasting Groups Method 108
The Borderline Group Method 112
Alternative Procedures and Limitations 113
The Body of Work and Other Holistic Methods 117
The Judgmental Policy Capturing Method 117
The Dominant Profile Method 120
The Analytical Judgment Method 121
Summary Analysis of Three Holistic Methods 122
Overview of the Body of Work Method 123
Procedures for the Body of Work Method 124
Training 125
Rangefinding 125
Pinpointing 129
Calculating Cut Scores 129
Interpreting the Logistic Regression Output 135
An Application of the Body of Work Method 138
Selecting Work Samples 139
Training Participants 140
Rangefinding 140
Alternative Procedures and Limitations 148
The Bookmark Method 155
Overview of the Bookmark Method 157
The Ordered Item Booklet 160
The Response Probability (RP) Value 162
Response Probabilities and Ordered Item Booklet Assembly-Rasch Model 162
Response Probabilities and Ordered Item Booklet Assembly-2PL Model 167
Directions to Bookmark Participants 172
Calculating Bookmark Cut Scores 176
An Implementation of the Bookmark Procedure 177
Training 177
Introducing the Ordered Item Booklet 179
Round One of a Bookmark Procedure 180
Obtaining Preliminary Bookmark Cut Scores 181
A Caveat and Caution Concerning Bookmark Cut Scores 184
Round One Feedback to Participants 185
Round Two of a Bookmark Procedure 186
Round Three of a Bookmark Procedure 187
Alternative Procedures and Limitations 189
The Item-Descriptor Matching Method 193
Procedures for the IDM Method 194
Alternative Procedures and Limitations 202
The Hofstee and Beuk Methods 207
The Hofstee Method 209
Procedures for Implementing the Hofstee Method 209
The Beuk Method 212
Procedures for Implementing the Beuk Method 212
Alternative Procedures and Limitations 214
Challenges and Future Directions in Standard Setting 217
Scheduling Standard-Setting Activities 219
Scheduling Standard Setting for Educational Assessments 219
Overall Plan 222
Participants 225
Materials 231
Final Preparations 234
At the Standard-Setting Site and Following Up 236
Scheduling Standard Setting for Credentialing Programs 237
Overall Plan 238
Participants 242
Materials 243
Final Preparations 244
At the Standard-Setting Site and Following Up 244
Conclusions and Recommendations 246
Vertically-Moderated Standard Setting 249
The Interrelated Challenges 250
A Brief History of Vertically-Moderated Standard Setting 253
What Is VMSS? 254
Approaches to VMSS 256
Applications of VMSS 257
An Illustration of VMSS Procedures 262
The Assessment Context 262
Preparing to Implement a VMSS Approach 263
Training VMSS Participants 264
Facilitating the VMSS Standard-Setting Meeting 265
Vertical Articulation of Cut Scores 266
Final Review, Adoption, and Conclusions 270
Alternative Procedures and Limitations 271
Standard Setting on Alternate Assessments 275
The Unique Challenges of Alternate Assessment 276
Necessary Conditions for Alternate Assessment Systems 278
A Generalized Holistic Method 283
Overview of an Application of the GH Method 285
Procedures for Implementing the GH Method 286
Conclusions and Discussion Regarding the GH Method 292
Alternative Procedures and Limitations 293
Special Topics and Next Steps 297
Rounding 298
Methods of Adjusting Cut Scores 299
Deciding How to Incorporate Uncertainty 302
Generalizability of Standards 306
Decision Consistency and Decision Accuracy 307
A Demonstration of Computing Decision Consistency and Decision Accuracy on Complex Tests 313
Other Decision Consistency Procedures 315
Summary and Future Directions 317
Using Multiple Methods of Standard Setting 319
Improving Participant Training 320
References 323
Glossary 333
Author Index 341
Subject Index 345
About the Authors 351
Subjects