Standard Setting: A Guide to Establishing and Evaluating Performance Standards on Tests » (1st Edition)

Michael B. Bunch is Vice-President of Measurement Incorporated, a test development and scoring company serving the statewide assessment community. Dr. Bunch joined MI in 1982 and has built the research and development division of that organization into a team of more than 60 project directors, psychometricians, editors, content specialists, and support staff. Prior to joining MI, Dr. Bunch was a senior professional with NTS Research Corporation, where he provided ESEA Title I evaluation technical assistance to state education agencies. From 1976 to 1978, he was a research psychologist with the American College Testing Program (ACT). Dr. Bunch has authored dozens of professional and technical reports on a variety of topics—many on standard setting—and has presented his work at national conferences, including annual meetings of the American Educational Research Association (AERA), National Council on Measurement in Education (NCME), American Psychological Association (APA), Council of Chief State School Officers (CCSSO), Education Commission of the States (ECS) and Adult Education Research Conference (AERC). He received his Ph.D. in psychological measurement from the University of Georgia.

In Standard Setting: A Guide to Establishing and Evaluating Performance Standards on Tests, authors Gregory J. Cizek and Michael B. Bunch provide the only "go-to" reference on how to set standards on tests in education, licensure, and certification. This book is comprehensive in scope, practical in nature, and definitive in terms of cataloguing the essential conceptual and procedural fundamentals of setting performance standards.

Key Features:

Offers a detailed description of procedures: In a practical, straightforward, and virtually jargon-free style, this book begins with important foundational and conceptual information about standard setting followed by step-by-step procedures for all currently used methods, including Angoff, Item Mapping, Analystic Judgment, etc. In addition, a full chapter is devoted to future issues and research in standard setting.

Enables readers to design, implement, and evaluate a standard setting study: Using this book, readers will be able to: select an appropriate standard setting method; schedule and plan implementation; identify and train qualified standard-setting participants; conduct the standard setting meeting; calculate one or more cut scores; and evaluate the success of the procedure they have implemented.

Allows readers to adapt written materials to their situation: Examples are provided in widely available formats such as Excel, SPSS, .pdf and Word files so readers do not have to purchase specialized software or learn to use unfamiliar packages. The book includes downloadable forms, sample correspondence, checklists, data sets, and analytical software for performing calculations required for various standard setting procedures.

Intended Audience:

This is an excellent supplementary textbook for graduate courses such as Applied Measurement in Education; Psychometric Theory; Advanced Educational Measurement; Special Topics in Measurement and Evaluation; Seminar in Educational and Psychological Measurement in the departments of Educational Psychology, Measurement and Evaluation, Psychology, and others.

Preface     xi
Fundamentals of Standard Setting     1
Contemporary Standard Setting: An Enduring Need     5
The Need to Make Decisions     6
The Benefits of Standard Setting     8
General Approaches to Standard Setting     9
Current Contexts for Standard Setting     11
What Is Standard Setting?     13
Kinds of Standards     14
Definitions of Standard Setting     14
Policy Issues and Standard Setting     19
Scoring Models     20
Research on Standard Setting     22
Rounding     23
Classification Errors     25
Item Scoring Criteria and Total-Test Performance Standards     29
Conclusions     33
Common Elements in Setting Performance Standards     35
Purpose     36
Choosing a Standard-Setting Method     41
Performance Level Labels     44
Performance Level Descriptions     46
Key Conceptualizations     48
Selecting and Training Standard-Setting Participants     49
Providing Feedback to Participants     53
Normative Feedback     54
Reality Feedback     55
Impact Feedback     56
Professional Guidelines for Standard Setting     57
Evaluating Standard Setting     59
Conclusions and a Foreword     63
Standard-Setting Methods     65
The Nedelsky Method     69
Procedures for the Nedelsky Method     70
Alternative Procedures and Limitations     71
The Ebel Method     75
Procedures for the Ebel Method     75
Alternative Procedures and Limitations     78
The Angoff Method and Angoff Variations     81
Procedures for the Angoff Method     82
Procedures for Angoff Variations     87
The Extended Angoff Method     87
The Yes/No Method     88
Alternative Procedures and Limitations     92
The Direct Consensus Method     97
Procedures for the Direct Consensus Method     98
Alternative Procedures and Limitations     102
The Contrasting Groups and Borderline Group Methods     105
Procedures for the Contrasting Groups Method     106
An Example Using the Contrasting Groups Method     108
The Borderline Group Method     112
Alternative Procedures and Limitations     113
The Body of Work and Other Holistic Methods     117
The Judgmental Policy Capturing Method     117
The Dominant Profile Method     120
The Analytical Judgment Method     121
Summary Analysis of Three Holistic Methods     122
Overview of the Body of Work Method     123
Procedures for the Body of Work Method     124
Training     125
Rangefinding     125
Pinpointing     129
Calculating Cut Scores     129
Interpreting the Logistic Regression Output     135
An Application of the Body of Work Method     138
Selecting Work Samples     139
Training Participants     140
Rangefinding     140
Alternative Procedures and Limitations     148
The Bookmark Method     155
Overview of the Bookmark Method     157
The Ordered Item Booklet     160
The Response Probability (RP) Value     162
Response Probabilities and Ordered Item Booklet Assembly-Rasch Model     162
Response Probabilities and Ordered Item Booklet Assembly-2PL Model     167
Directions to Bookmark Participants     172
Calculating Bookmark Cut Scores     176
An Implementation of the Bookmark Procedure     177
Training     177
Introducing the Ordered Item Booklet     179
Round One of a Bookmark Procedure     180
Obtaining Preliminary Bookmark Cut Scores     181
A Caveat and Caution Concerning Bookmark Cut Scores     184
Round One Feedback to Participants     185
Round Two of a Bookmark Procedure     186
Round Three of a Bookmark Procedure     187
Alternative Procedures and Limitations     189
The Item-Descriptor Matching Method     193
Procedures for the IDM Method     194
Alternative Procedures and Limitations     202
The Hofstee and Beuk Methods     207
The Hofstee Method     209
Procedures for Implementing the Hofstee Method     209
The Beuk Method     212
Procedures for Implementing the Beuk Method     212
Alternative Procedures and Limitations     214
Challenges and Future Directions in Standard Setting     217
Scheduling Standard-Setting Activities     219
Scheduling Standard Setting for Educational Assessments     219
Overall Plan     222
Participants     225
Materials      231
Final Preparations     234
At the Standard-Setting Site and Following Up     236
Scheduling Standard Setting for Credentialing Programs     237
Overall Plan     238
Participants     242
Materials     243
Final Preparations     244
At the Standard-Setting Site and Following Up     244
Conclusions and Recommendations     246
Vertically-Moderated Standard Setting     249
The Interrelated Challenges     250
A Brief History of Vertically-Moderated Standard Setting     253
What Is VMSS?     254
Approaches to VMSS     256
Applications of VMSS     257
An Illustration of VMSS Procedures     262
The Assessment Context     262
Preparing to Implement a VMSS Approach     263
Training VMSS Participants     264
Facilitating the VMSS Standard-Setting Meeting     265
Vertical Articulation of Cut Scores     266
Final Review, Adoption, and Conclusions     270
Alternative Procedures and Limitations     271
Standard Setting on Alternate Assessments     275
The Unique Challenges of Alternate Assessment     276
Necessary Conditions for Alternate Assessment Systems     278
A Generalized Holistic Method     283
Overview of an Application of the GH Method     285
Procedures for Implementing the GH Method     286
Conclusions and Discussion Regarding the GH Method     292
Alternative Procedures and Limitations     293
Special Topics and Next Steps     297
Rounding     298
Methods of Adjusting Cut Scores     299
Deciding How to Incorporate Uncertainty     302
Generalizability of Standards     306
Decision Consistency and Decision Accuracy     307
A Demonstration of Computing Decision Consistency and Decision Accuracy on Complex Tests     313
Other Decision Consistency Procedures     315
Summary and Future Directions     317
Using Multiple Methods of Standard Setting     319
Improving Participant Training     320
References     323
Glossary     333
Author Index     341
Subject Index     345
About the Authors     351