Arnold GK, Guille R, Lipner RS. — American Board of Internal Medicine
Presented: International Conference on Questionnaire Design, Development, Evaluation, and Testing (QDET2), November 2016
Abstract: We developed a sample adequacy criteria (SAC) for assessing sample quality of physician volunteers in community engagement activities related developing certification examinations at the American Board of Internal Medicine (ABIM). SAC aids judgments whether volunteer exam reviewers actually represent the expected views of the eligible physician pool invited to review or whether additional resources are put into data collection.
SAC uses three benchmarks for judging sample fit for use. One benchmark is defined such that the number of volunteers meets or exceeds 220 and 12 or more volunteers review each section of the exam material. This benchmark serves the power and reliability requirements for the tasks. The second benchmark compares sample auxiliary measures to those of the pool of eligible reviewers using standardized differences. Auxiliary variables include geographic region, sex, age and percentage practice time in patient care.
The third benchmark uses sample representativeness indices including the “representativeness” R index, “balance” B index, the propensity score coefficient of variation, and propensity score correlations with auxiliary variables. Other metrics come from comparing the volunteer sample with bootstrap samples of non-respondents of equal size to assess balance among auxiliary variables. If there is sufficient bias, correction with weighting is made. The SAC assumes if sample estimates of the population auxiliary variables are unbiased, then so are sample ratings used in the exam activities. The assumption is later verified by comparing correlations between auxiliary variables with actual exam-related ratings.
SAC is used to make one of three determinations: 1) use the sample without weighting or 2) use the sample with weighting or 3) the sample is not adequate so further data collection is required. SAC has been used in volunteer reviews of seven Maintenance of Certification exams for medical subspecialties. Exemplar SAC results from these exams and validating information for the process are presented.
For more information about this presentation, please contact Research@abim.org.