"What Do They Want Me To Say?" The hidden curriculum at work in the medical school selection process: a qualitative study

Abstract

Background

There has been little study of the role of the essay question in selection for medical school. The purpose of this study was to obtain a better understanding of how applicants approached the essay questions used in selection at our medical school in 2007.

Methods

The authors conducted a qualitative analysis of 210 essays written as part of the medical school admissions process, and developed a conceptual framework to describe the relationships, ideas and concepts observed in the data.

Results

Findings of this analysis were confirmed in interviews with applicants and assessors. Analysis revealed a tension between "genuine" and "expected" responses that we believe applicants experience when choosing how to answer questions in the admissions process. A theory named "What do they want me to say?" was developed to describe the ways in which applicants modulate their responses to conform to their expectations of the selection process; the elements of this theory were confirmed in interviews with applicants and assessors.

Conclusions

This work suggests the existence of a "hidden curriculum of admissions" and demonstrates that the process of selection has a strong influence on applicant response. This paper suggests ways that selection might be modified to address this effect. Studies such as this can help us to appreciate the unintended consequences of admissions processes and can identify ways to make the selection process more consistent, transparent and fair.

Background

Admission to medical school is a serious matter for all parties involved. The task of selecting applicants who wish to become doctors is "a complex problem encumbered by inherent uncertainty" [1], and "daunting... selecting a small number of applicants who are most likely to succeed in medical school from a large pool of seemingly suitable applicants"[2]. Traditionally, selection has been based largely on academic performance, but recent work has focused on making the process more effective and representative of the needs of society [3, 4] and has encouraged admissions committees to consider non cognitive attributes as well as academic performance [5].

Many medical schools utilize essay questions in student selection, and the essay question has long been included as a component of the Medical College Admission Test (MCAT). Literature on the use of the essay question in selection provides little consensus on its role; some suggest there is little evidence to support the continued use of essays [6], while others point out that essays complement selection based upon academic performance [7]. There is evidence that performance on an essay question reflects post-entry performance in applicants entering health-related professions [8], and that admissions essays scored for writing skills, clarity, organization and cohesiveness can predict academic success, and identify "problem students"[9]. A qualitative analysis of nursing school admission essays demonstrated that essays could differentiate between students who would complete their studies from those who would not, and that the essay question was a better predictor than measures of prior academic performance [10]. It has also been suggested that writing skills should be included in admissions processes because they are an essential part of healthcare education and practice [11, 12].

The essay has been part of the selection process at our school for many years, being submitted along with transcripts, MCAT scores, and letters of reference to determine whether an applicant would be invited for interview. With the adoption of the multiple mini interview at our school [13, 14], we also added two essays written in an invigilated setting. This study builds upon previous work in which we analyzed how applicants answered essay questions used in selection at our medical school [15], and was intended to provide a further exploration of observed patterns of applicant response. In that study, we concluded that applicants approached the essay questions as a way to tell their own subjective story; this approach was different from what was intended and expected by the school. The intent of this further study was to gain a better understanding of how applicants approached the essay questions utilized and to situate the findings in the context of the selection process as a whole.

Methods

This study was based on the analysis of three essay questions utilized in the selection process at our school in 2007. The school is a three-year medical school which received a total of 1,476 applications in the year under study. From this group, 417 students were invited for interview, and 125 students were finally offered admission. Questions were developed by the Admissions Committee and were intended to test for traits of altruism, understanding of professionalism, and reflection/self-assessment (Table 1). The essay on volunteerism was submitted online before interview, with the question being posted three months before the application deadline, while the other two essays, on professionalism and reflection, were completed by applicants at the conclusion of the multiple mini-interview process. Each essay was approximately one page in length and was completed in 30 minutes. The online essay was scored by four reviewers, while the essays done at the time of interview were marked by a single assessor and made up a total of 10% of the score used for final ranking. No other writing by the applicant (eg. a personal statement) was included in the selection process.

Table 1 Essay Questions & Questions used in Interviews with Applicants and Assessors

Essays for analysis were selected at random from the applicants invited for interview, and were anonymized before analysis commenced. The analysis was undertaken independently of the admission decisions process. All applicants gave permission for use of their written submissions for research purposes.

As this area of the literature is relatively under-examined, we chose to conduct an exploratory, qualitative approach to analyze the essays in order to derive a conceptual framework which would describe the patterns of response observed, and to express the relationships between the ideas and concepts encountered [16–18]. As our previous work has focused on the content of applicant responses ("what was said") [15], this study was intended to examine the patterns of applicant response in more detail ("how it was said") with the intent of determining the factors which influence applicant response ("why it was said").

In keeping with the qualitative methodology employed, it is important to note the perspective and intent of the study team involved in this work. The study was not designed to consider the value of the questions themselves, nor how they were created, or to develop "better" or "more objective" questions; this was the reason we did not adopt an approach based on linguistics or hermeneutics. Neither did we plan to apply a psychometric approach to determine the reliability or validity of the questions used in selection. Instead, the conceptual basis of this study was to observe an existing complex process in situ, with the aim of reaching a better understanding of how applicants approach selection for medical school, and then moving beyond the immediate context to situate these findings in the larger context of the admissions process as a whole.

Analysis was first conducted on 60 examples of each of the three essays selected at random, focusing on the patterns of writing observed in each essay. Essays were analyzed by a two readers (JW and JL) using iterative reading of texts, and selection of key words and phrases to develop a coding scheme in which the ideas expressed were categorized. Reflective journaling, charting and memo-writing were also used to record observations on the underlying themes in the data as analysis proceeded. Coding was reviewed, coding structures amended and debated, and consensus achieved at data analysis meetings held throughout the study period, and a theoretical explanation of the observed patterns of applicant response was generated. At the conclusion of analysis, 10 more examples of each essay were analyzed to ensure that data saturation (no further emergent themes) was achieved. The findings of this study are thus based on analysis of a total of 210 essays (60 + 10 for each of 3 essays).

After completion of analysis, we held interviews with a number of successful applicants and assessors who were involved in the admissions process in 2007. Convenience sampling was used; all applicants invited for interview had applied to the University of Calgary, and were offered admission either to the University of Calgary or the neighbouring University of Alberta. No unsuccessful applicants were contacted. Applicants were invited to participate by email; sessions were held with 20 applicants in two groups (60 minutes each) and with 4 assessors one-on-one (30 minutes each), and were tape-recorded and transcribed. We tested the generated theory and discussed the findings of analysis with these participants to gain another perspective on the role of the essay in the admissions process (participant checking/data triangulation). Questions used to stimulate discussion were generated from the results of analysis and are shown in Table 1. Ethical approval for the study was obtained from the University of Calgary Conjoint Health Research Ethics Board.

Results

Analysis of the essays revealed that in many instances, material given in an answer did not seem to relate directly to the question provided, but appeared to have been included for another, unstated purpose. Answers included information which did not bear directly on the question asked, but seemed instead to be intended to impress the reader. For instance, in the question on volunteerism: "my GPA increased dramatically in the subsequent two years, ending with a 3.78/4 for my final year". Such examples were coded as "impressive details" and "impressive stories". Applicants also used their answers to express a sense of entitlement to admission of medical school (coded as "expressing expectations of reward") and to state that they were sure they were good enough to be admitted ("statements of confidence and destiny"). There were also many examples of applicants answering a question by restating and agreeing with the assumptions implicit in the question, as distinct from considering all aspects of the question asked (coded as "agreeing with the assumptions of the question"). Codes and representative quotations are presented in Table 2.

Table 2 Codes and Representative Quotations

Further analysis focused on describing and explaining these patterns of response in more detail, with the purpose of understanding the factors which applicants considered when determining how to answer these essay questions. This process led to the development of a theoretical explanation named "What do they want me to say?" based upon the patterns of response observed in the data.

Elaboration of theory

This theory proposes that considerations of "who will be reading my answer, and what do they expect me to say?" are at least as important to applicants as the content of the question itself. These effects are predicted to be especially relevant for a question with an "obviously correct" answer, in which applicants can easily divine and provide the "desired" response. These findings bring to light tension between "a genuine response" and "the expected response" that we believe applicants experience when choosing how to address an essay question in the admissions process. This point was probably best articulated in the following extract in which an applicant reflected on the difficulty of choosing between a "hypothetical and ideal protocol" (expected answer) and an "informal, genuine attitude" (a genuine answer): "Being graded on an interaction can be both a challenging experience and a rewarding one. It can be challenging in the sense that there is slight ambiguity as to how much one should try to follow some hypothetical and ideal protocol and how much one should maintain an informal and genuine attitude at the risk of compromising the latter formal aspect of a formal interview"

We propose two main factors which interact to determine how applicants decide "what do they want me to say?" when faced with a question in the selection process: 1: perceived expectations of the selection process and 2: ways of conforming to meet those expectations. We propose that expectations of the selection process are formed by obtaining information from other applicants, medical students and schools, and from the experience of prior application. We suggest that applicants consider a perceived 'acceptable' range of response, and ensure that their own response falls within this range. We detected much evidence that applicants include material expressly intended to enhance their chances of a successful admission to the school. This is considered to be the reason why applicants included statements of confidence and destiny ("I am ready"), and why they included impressive details ("I have a GPA of 4.0"). We also observed that most applicants did not include material likely to be perceived as unacceptable in the context of the selection process--no applicants gave answers disagreeing with the assumptions of the questions asked, suggesting some degree of self-censoring. As part of the theory generated, we also proposed that applicants would be expected to consider the behaviour of other applicants when deciding how to respond to a particular question.

Confirmation of theoretical explanation by participant checking/data triangulation

The elements of "What Do They Want Me to Say?" were supported by interviews with applicants and assessors (Figure 1 and Table 3). Applicants expressed a clear, shared understanding of the 'proper' answers expected by the school, and described modulating their responses to conform with their perceived expectations of the selection process. One applicant described her understanding of the school's expectations very succinctly: "I did that constantly through the entire interview process, thinking: "What would they want me to say?" One applicant described applying for medical school and "getting sliced and diced" because he did not understand "the unwritten rules" of the process in the same way as the other applicants. There was also discussion about the need to "sell yourself" in the application process, using terms such as "standing out" and "making your answer different". Applicants admitted behaviors such as "exaggerating" and "polishing", and gave accounts of having prepared answers, and repeating stories perceived to be effective. At interview, applicants described these behaviors as acceptable practice. Assessors also expressed concerns about exaggeration, and described essays which appeared to have been written with the clear intent of impressing the assessor.

Figure 1

The theoretical explanation "What Do They Want Me to Say?".

Table 3 Quotations from Applicants & Assessors

Discussion

This work demonstrates that that the process of selection has a strong influence on how applicants respond to questions used in medical school admissions. Applicants were observed to be conforming to the expectations of the selection process in ways intended to maximize their chances of acceptance. Applicants admitted to giving answers perceived to be the most acceptable to the school, and to having a shared understanding about the "unwritten rules" of acceptable conduct in the process. These findings suggest the existence of a "hidden curriculum" at work in the admissions process, by which applicants understand how they must behave to gain entry. This parallels a similar phenomenon previously noted in premedical education and beyond [19–23] which has been described as "a set of influences that function at the level of organizational structure and culture", "commonly held understandings... and taken-for-granted aspects of what goes on", "implicit messages... about what the institution considers important" and "implicit rules (used) to survive the institution" [21, 24]. In this case, we propose a set of influences arising from the selection process through which applicants gain an understanding of what the school desires and how applicants must behave in order to gain admission. Some of the "unwritten rules" of this hidden curriculum may be similar to those expressed by applicants in this study: "figure out what they want to hear", "sell yourself--make them want you", "exaggerate but don't get caught", "do whatever you have to do to get in".

Gaining admission to medical school is highly competitive: pre-medical education has been described as "a series of obstacles to be overcome on the way to the elusive goal of medical school admission" [19], with applicants possessing "a fiercely contentious desire to prevail over others, and at all costs get into this or that or any medical school" [25]. As pointed out in a recent review, this competitiveness has an effect on the selection process: "Trust no-one. Do not trust the applicants. When stakes are high, the likelihood of cheating increases; the stakes for medical school admission are very high" [26]. Others have observed medical school applicants "buffing up the application form" [27], and similar phenomena have been noted in the selection process for residency [28].

A number of theories of human psychology are relevant to our findings. Social learning theory suggests that applicant response will be determined by consideration of expectations, consequences, rewards and punishments occurring within a complex social system [29]. Theories of impression management and self-presentation also relate to our work, as they describe goal-directed conscious or unconscious processes in which people attempt to influence the perceptions of others by regulating and controlling information in social interaction [30]. Work on social desirability bias also bears on our findings, as it describes the tendency of respondents to reply in a socially-acceptable manner that will be viewed favorably by others, instead of responding with their own opinions, a phenomenon known as opinion conformity [31]. Applicants may infer from a question a socially-desirable "expected response" with which they are required to conform in order to obtain reward (admission to medical school) [32]. Conforming to the expectations of the application process may well be the first step in a longer, more sustained process of conforming which occurs throughout medical school and beyond. As Coles states: "we let (students) know by our admissions practices what kind of person we desire" [25]. It is interesting to speculate that problems relating to unprofessional behaviour may originate in messages sent to applicants at their entry into medical education [33, 34].

It can be argued that behaviours such as "exaggeration" and "selling" are simply an accepted part of any competitive selection process which operates "in the real world", although such behaviors have not been acknowledged in the literature on medical school admissions to date. It may be that we expect applicants to be "responsible for knowing when (and how) to break the rules" [35]. Such an approach creates inherent unfairness and bias against those who are operating under different assumptions, and creates a dissonance between what schools intend and what is experienced by applicants [36]. It is important to remember that applicants are only responding to the pressures of selection created by the processes that schools have chosen to employ [32, 36].

Selection might be improved by taking steps to make the "hidden curriculum of admissions" more explicit [23]. Addressing pre-medical culture and preparation is important [19], and we suggest the selection process should include discussion about applicants' perceptions of the pressures of the process, and about the conflict between needing to "sell yourself" while trying to remain genuine [23]. Steps such as these may assist in demystifying what is a high-pressure process, and may reduce anxiety and dispel incorrect beliefs about the process obtained from the "admissions rumor mill" [3, 20, 28, 32, 34, 36]. The nature of the questions used in selection is also important: other studies have suggested that self-declared values should not be used in admissions decisions [37], a finding very much in keeping with our core variable "What do they want me to say?" Answers to questions about the self should be treated with much caution in the selection process; as Siu and Reiter advise: "Avoid self-reporting... to expect the bad apples to weed themselves out on the basis of self-reporting goes beyond Pollyanna into self-delusion. All observations about the candidate should be made from a more remote position than the applicants themselves" [26].

This study was limited as it considered the selection process at only one school in one year, it examined essays written both online and in person, and involved only a subset of applicants and assessors in interview. We hope that our findings will spur other investigators to consider this under-represented part of the literature, and to determine if these effects can be observed elsewhere.

Our observations on the effects of the selection process are expected to be provocative, as they reveal a side of the admissions process which appears well-understood by applicants but which has not been reported in the admissions literature to date. We anticipate that this study will make uncomfortable reading for those engaged in the admissions process, as it considers the acceptability of the behaviours which are required to gain entry to medical school. As Hafferty has written: "until we come to accept that medical training is, at root, a process of moral enculturation and... medical schools function as moral communities, the reform that is needed--the reform that the public deserves--will remain both elusive and enigmatic" [21].

Conclusions

This work suggests the existence of a "hidden curriculum of admissions", and demonstrates that the process of selection for medical school has a strong influence on applicant response. Our findings also highlight the social context of the admissions process, with applicant response being affected by factors such as expectations, consequences, rewards and punishments. We suggest two ways in which the selection process might be modified to understand and address this effect: discussing the hidden curriculum of the admissions process more openly, and avoiding questions which ask directly about the self. Studies such as this can help us to appreciate the unintended consequences of admissions processes and can identify ways to make the selection process more consistent, transparent and fair.

Authors' information

Dr Jonathan White MD PhD is an Associate Professor and Tom Williams Endowed Chair in Surgical Education in the Department of Surgery at the University of Alberta. Dr Keith Brownell MD is a Professor in the Departments of Clinical Neurosciences and Medicine at the University of Calgary. Dr Jean-Francois Lemay MD is an Associate Professor in the Department of Pediatrics at the University of Calgary. Dr Jocelyn M Lockyer PhD is a Professor in the Department of Community Health Sciences and is Associate Dean for Continuing Medical Education and Professional Development, University Of Calgary.

Ethical approval

University of Calgary Conjoint Health Ethics Review Board.

Published in abstract form

White JS, Lockyer J. "What Do They Want Me to Say?" A Qualitative Analysis of the Use of the Essay Question in Medical School Admissions (abstract), Medical Education 2009, 43(Supp1): 27