This study represents the largest and only multi-centered study to date of inaccuracies on residency applications. Review of 28.6% of the total applicant pool revealed an error on 11.6% of all applications. This is similar to those described in prior reports in EM as well as other specialties [1–8]. If benign errors are excluded, the rate of major errors on applications is nearly 10%, which would place EM among the lowest of reviewed fields.
Our study design was mandated by features of the application system. For example, there is no centralized way to review applications, so a multi-centered trial was necessary, despite the duplication of effort this entails. In addition, the rules governing the behavior of programs during the NRMP match make discussion of applicants between residencies difficult, so programs are often reluctant to share negative information about applicants. This forced our study towards a design that would maximize protection of the applicant, while utilizing the available pathways for reporting of errors and for corrective action. In addition, ongoing legal efforts necessitated legal consultation with a lawyer assigned to the defense of Jung vs. Association of American Medical Colleges in order to ensure protection of the residency programs involved.
A prior retrospective study at one of the authors' institution (EDK) showed similar findings with a few notable differences. We found no erroneous AOA claims while Roellig[7] found an error rate of 35.7%. It is possible that this is due to the differences in included applications: the prior study included osteopathic students while ours did not. As three of the five erroneous AOA claims in that study were from osteopathic students, it is possible that this was due to students attributing this moniker to the American Osteopathic Association.
Secondly, our study found a higher rate of errors on publications (41.0% vs. 21.3% in Roellig[7] and 20.4% in Gurudevan[8]) than previously reported. There are many conceivable sources for this discrepancy. It is possible that the literature search methodology was insufficient in our project, as it relied on review by database, rather than journal review. Before further study is begun, time from publication to database entry must be assessed. In addition, the prior, retrospective, studies were able to assess journals listed as "in press," which was not possible with a prospective design, due to the time interval between acceptance, revision and publication. The inter-rater reliability implies that the methodology provided consistent results, but the disparity between our results and previous results from our field is concerning.
A third difference is that the prior study[7] found 73.3% of advanced degree claims to be verifiable while we were able to verify only 56.9%. Several institutions contacted by the authors had procedures that prevented verification. The previous, retrospective, trial could invest the time necessary to address these procedures, while a prospective, multi-centered trial could not.
Lastly, during the Roellig trial[7], we were able to confirm the AOA status of applicants up to 8 months after the application season. During the current project, the AOA offices were less able to accommodate our inquiries. Should this project become much larger, the AOA staff may become reluctant to verify applicant election to AOA. However, the MSPE's were uniformly helpful in confirming AOA status. It should be noted that we chose not to validate claims of AOA nomination as that process is more loosely defined, and other than each applicant's dean's office, there is not a reliable avenue for verification (the AOA database does not track nominations).
In our study, members of the research program did not reliably take corrective action when an inaccuracy was found. Because of this, we are unable to draw conclusions as to the efficacy of the reviews. If this persists, it may limit the effect of the project. This matter will be strongly emphasized before the next application season.
We did not attempt to assess whether the inaccuracies were intentional though we did attempt to differentiate "benign" and significant errors. For instance, typographical errors may be reasonably assumed to by unintentional whereas an erroneous publication or advanced degree claim is more likely assumed to be intentional. Others inaccuracies may be more difficult to assess. For instance, the misrepresentation of the order of authorship of an article could be assumed to be intentional or simply due to a lack of understanding regarding the significance of the order.
Since the interpretation of motivation is fraught with difficulty, the study did not make any recommendations for how each program used the findings on its applicants. In addition, we did not seek to recommend specific action by any dean's office or the NRMP. Decisions of this nature were outside our scope of research, but may prove to be fertile ground for future investigation.
During the next application season, we intend to expand our study to other sites, with a goal of eventually reviewing the majority of applicants to our specialty. In addition, we will publicize the existence of the trial to the applicants before their applications are submitted. In future studies, we aim to impact the number of erroneous claims. By notifying applicants' institutions and the NRMP regarding inaccuracies and publicizing these efforts, we anticipate a decrease in intentional erroneous claims on applications.
