Carr J, Guille R, Lipner R, Taylor K. — American Board of Internal Medicine
Parshall C. — Measurement Consultant
Presented: Association of Test Publishers Annual Meeting, February 2013
Abstract: In this session, we discuss the need to manage change when implementing a variety of assessment innovations. We describe a systematic and wide-ranging approach to including innovative item types and other fidelity enhancements across a large number of exam programs. Implementing these fidelity innovations requires organizational, procedural and technical changes to support the overall change in direction. We discuss strategies to stimulate organizational change with respect to test development when assessment innovations are included. These include top-down approaches that help clarify goals and priorities for senior management; bottom-up approaches that encourage measurement staff to act as catalysts for change; support for cross-functional teamwork; managing geographically distributed teams; handling negativity and overzealousness in the group process (e.g., blockers, dominators and special interest pleaders); and conflict resolution. Next, we present an overall procedural change for implementing test innovations. This rollout plan for innovations includes an extensive sandbox (i.e., exploratory) stage. We discuss the wide range of feasibility analyses that can be beneficially carried out during sandboxing–such as evaluating whether the potential innovation is likely to improve fidelity to practice, and whether there are psychometric implications for test form assembly or alternative scoring methods. We then discuss the technical changes frequently required when new item types or other fidelity enhancements are implemented. In a typical test development process, items must pass through multiple distinct software systems–including item banking, test publishing, test delivery, test scoring, item analysis and reporting. Given this situation, it is not surprising that software support is often incomplete or absent. The implications of this, and possible approaches, are considered. Finally, we discuss the use of research in order to set the direction of change. Various approaches to research are discussed, from simple surveys of exam committees regarding the fidelity of innovations they find most content relevant, to extensive analyses of more substantial changes to exam delivery. Graphical methods for summarizing a set of cost-benefit analyses are illustrated.
For more information about this presentation, please contact Research@abim.org