They participated in several rounds of practice and were given feedback prior to making operational ratings. Raters were trained to use a standardized rating process and scoring rubrics. Data from multiple years must be combined in order to report and interpret results. *While we have begun to collect performance data from participating applicants in the ERAS 2018 cycle, sample sizes are too small to draw any conclusions. We provided data in some areas (i.e., psychometrics) immediately following each SVI pilot year and are still waiting for data to accumulate in other areas (e.g., correlations with performance). As shown in the table below, data from multiple ERAS application cycles were used to evaluate the broad areas. Each broad area was composed of multiple research questions. The SVI evaluation plan included four broad areas: (1) psychometrics, (2) validity evidence, (3) fairness and preparation, and (3) program director and applicant reactions. For more detailed information, refer to cited articles. This document provides an overview of the AAMC’s evaluation of the SVI. As such, the AAMC decided not to renew or expand the SVI pilot in emergency medicine for the ERAS 2021 application cycle.
However, there is a lack of interest among the emergency medicine community in continuing to use and research the SVI, as well as operational challenges scaling the SVI to the full applicant pool across multiple specialties. The AAMC stands confidently behind the SVI as a reliable, valid assessment of behavioral competencies that does not disadvantage individuals or groups.
At the conclusion of the SVI for ERAS 2020 season, the AAMC reviewed data from the last four years.