Becker K. — Pearson VUE
Bovell D, Guille R, Lipner RS. — American Board of Internal Medicine
Presented: Association of Test Publishers Annual Meeting, February 2011
Abstract: Computerized adaptive Multistage Testing (Ca-MST) is a test design that administers a “panel” of items in an adaptive fashion, allowing examinees the opportunity to traverse the panel in the pathway that is best tailored to their particular ability levels and to terminate early in particular situations. Since it is administered in pre-assembled sets of items, called “testlets,” Ca-MST allows content experts to carry out a form-level content review. While the test structure may be a better fit than computer adaptive testing (CAT) for many current forms-based testing programs, there are currently few programs using Ca-MST. This session will provide a practical look at transitioning from forms-based testing to multistage testing. Panelists from a certification board and a test delivery vendor will discuss issues in transitioning the board from a linear computer-based test to an adaptive, multistage testlet-based examination. We will discuss our experience in “selling” the concept to leadership in the organizations and the research effort required to develop a design that achieves two unrelated goals: (1) permitting the majority of highly able examinees to terminate early, while (2) retaining or even improving the precision of diagnostic feedback to failing examinees. Topics that will be discussed include: (1) organizational commitment to innovation and overcoming obstacles to achieve innovation; (2) strategies for automated testlet assembly; (3) real-time scoring using Item Response Theory (IRT), (4) administrative issues and (5) communication about the change to stakeholders including physicians, patients and health care institutions. Attendees will take away an understanding of the rationale for multistage testing, the advantages and disadvantages involved in transitioning from linear computer-based testing and the psychometric issues in fitting multistage testing to a form-based program.
For more information about this presentation, please contact Research@abim.org.