Zhang Y, Zhu R. — American Board of Internal Medicine
Presented: American Educational Research Association Meeting, April 2011
Abstract: Aberrant response patterns occur when examinee item responses are inconsistent with their abilities. Aberrant item responses due to prior knowledge or other types of security breach are of particular concern for high-stakes tests because they lead to inaccurate estimation of examinee ability and faulty classification decision, and impact the reliability and validity of test scores adversely. The objective of this study is to explore the feasibility of using a person-fit statistic for identifying aberrant response patterns on multiple choice items in the context of a medical certification examination. Both simulated and empirical data were used in the study. The simulated data modeled aberrant behaviors due to improper prior knowledge in the low ability and borderline groups. The results from the simulation study showed high true detection rates and low false positive detection rates under two separate conditions: 1) The borderline ability group getting easy items wrong, and 2) The low ability group getting difficult items correct. The true detection rate was low and the false positive detection rate was high for the borderline ability group getting the difficult items correct. The z statistic did not seem to be very useful for detecting examinees with prior knowledge of test items when applied to empirical data. Future directions of research were discussed in the paper.
For more information about this presentation, please contact Research@abim.org.