Song H, Sares TA, Zhu R, Lipner RS. — American Board of Internal Medicine
Presented: National Council on Measurement in Education Meeting, April 2011
Abstract: Accuracy and consistency of pass-fail decisions are of primary interest to high-stakes certification testing programs. For medical certification programs, in particular, decision accuracy and consistency may affect patient safety as well as a physician's career advancement. This study evaluates the applicability of Rudner's decision accuracy and Huynh's decision consistency methods for ABIM certification and Maintenance of Certification (MOC) examinations during transition from classical test theory (CTT) to item response theory (IRT) scoring. The pass-fail decision accuracy and consistency were determined from simulated true and estimated abilities and provided a nonparametric method to cross-validate those model-based estimates. Results verified that Rudner's IRT-based decision accuracy procedures are convenient, efficient and “in line” with IRT scoring classification logic. Results also supported the proposition that the Huynh procedure is applicable for estimating pass-fail decision consistency for exams in the IRT framework via a simple conversion between the IRT and the raw score standards.
For more information about this presentation, please contact Research@abim.org.