Traditionally, onsite interviews functioned as high-stakes events that substantially affected an applicants’ final rank ordering of postgraduate surgical training programs [1, 2]. Unfortunately, the COVID-19 pandemic forced programs to suspend in-person interviews within the United States [3,4,5,6]. Surgical fellowship programs were among the first to transition interviews to a virtual format, by default becoming a testing ground for other surgical training programs. Prior to COVID-19, the feasibility and utility of virtual interviews for postgraduate programs were assessed with varying results. Some studies found an overall applicant preference for on-site versus online interviews [7,8,9,10], while faculty surveys demonstrated a desire towards utilizing virtual interviews to potentially screen candidates before in-person interviews [8, 9].

During the transition to virtual interviewing, multiple guides on interview preparation and conduct were published in 2020 for both programs and candidates [3,4,5, 11, 12]. The majority focused on the technical aspects of virtual interviews (software recommendations, ideal interview environment, communication strategies, etc.) with little attention paid to how applicants experienced the recent transition to a virtual platform. We were motivated by anecdotal reports of perceived suboptimal interview experiences from residents training in our institutions. To determine unique challenges associated with the new interviewing format, we completed a qualitative analysis of recent interviewee experiences. In addition to analyzing their perceptions on the process, we compiled a list of their recommendations to faculty and applicants for future fellowship matches.

Methods

Study design

A thematic qualitative analysis rooted in grounded theory was conducted. We explored the challenges of the virtual interview process for applicants participating in the Fellowship Council’s (FC) 2020–2021 fellowship year application and matching process. The FC is an association of program directors and specialty societies who oversee over 170 surgical fellowship training programs including: Advanced Gastrointestinal (GI), Advanced GI Minimally Invasive Surgery (MIS), Bariatric, Flexible Endoscopy, Foregut, Hepatobiliary (HPB), Thoracic, and non-ACGME-Accredited Advanced Colorectal Fellowships. Fellowship interviews for the 2020–2021 FC match were conducted between February–May 2020.

Semi-structured interviews by telephone or video-conferencing software were the primary source of data. All interviews were conducted one-on-one by a male research fellow with formal education in qualitative interviewing (RA). This interviewer is junior to the postgraduate level of study participants, has no supervisory or evaluative role over participants, and no professional connection to the FC. Participants were informed that their responses were confidential. An interview guide was developed from a literature review and in consultation with subject and methodology experts. The guide included open-ended questions regarding the challenges of interviewing virtually for surgical fellowship programs and solicited suggestions for program directors and future applicants on how to improve the remote match process. The interview guide was piloted on a co-investigator, an applicant of the 2020 surgical fellowship match (SM), and refined based on feedback. The pilot interview was not included in the analysis. The study was approved by the Partners Healthcare Institutional Review Board (Protocol #: 2020P002845) of Massachusetts General Hospital (MGH) and followed the Consolidated Criteria for Reporting Qualitative Research (COREQ) reporting guidelines [13]. A COREQ checklist is provided in the Supplemental Material.

Participants

All registered applicants to the FC’s 2020–2021 fellowship year application and matching process, as well as applicants to non-FC fellowships from our institution (MGH), were invited to participate in the study. FC applicants were recruited via an email sent by the FC, and MGH participants recruited via an email from a member of our research team. The emails were sent in November 2020, after participants had received their match results. A purposive criterion-based sampling technique was used. Forty-one applicants agreed to participate in the study and completed a brief demographic survey. Of these 41 initial respondents, 20 were selected by pre-defined criteria aimed to maximize the sample’s diversity based on: gender, race/ethnicity, current training center (academic or community), geographic region, and fellowship specialty. Of the 20 applicants, not all were selected as study participants. Interviews were scheduled per the applicants’ convenience and conducted until thematic saturation was reached and the cohort was balanced to the best possible degree. Saturation was determined when new themes emerged infrequently, and code definitions remained stable. All participants verbally consented to participate at the beginning of each interview.

Data collection and analysis

Interviews were recorded and transcribed verbatim by a research fellow (RA), and all identifying personal information was removed. Transcripts were independently inductively coded by two research fellows (RA, RDS). A codebook with detailed definitions was developed and iteratively modified using the constant comparative method. Codes were applied to each transcript, and the data was organized using Dedoose™ data management software (version 8.1.8, SocioCultural Research Consultants, LLC, Los Angeles, CA). Coding conflicts were resolved through discussion and, if the two independent coders were unable to reach agreement, a third researcher evaluated for consensus (DG). The majority opinion was coded and reported in the final analysis. Coding continued until inter-coder agreement (Cohen’s kappa) reached > 0.8. Primary codes were then collapsed into coding categories and analyzed for broader themes. Following the completion of the coding process, member checking was completed by sharing emergent themes and associated definitions with a convenience sample of two prior applicants. No further changes to the themes were made based upon this feedback. Multiple peer debriefings were conducted to discuss and agree upon the emerging themes.

Results

Fifteen surgical trainees from 15 different institutions were interviewed to reach data saturation. The final intercoder kappa coefficient was 0.95, indicating strong agreement. The median participant age was 32.5 (range 30–45) and 60% (n = 9) were of male gender. Mean interview length was 22.8 min (standard deviation 5). More than half of the participants (55%) had applied for a fellowship in Advanced GI and MIS surgery, 30% in Bariatrics, 10% in HPB, and 5% in Surgical Oncology. Participants attested to having interviewed for a median of 13 fellowship programs (range 1–31), of which a median of 70% were virtual (Table 1).

Table 1 Demographic characteristics
Full size table

From the 18 primary codes, four major themes emerged, describing the challenges associated with the virtual interview process: (a) perceived data deficiency, (b) superficial personal connections, (c) magnification of non-professionalism, and (d) logistical frustrations (Fig. 1, Table 2).

Fig. 1
figure 1

The challenges of virtual interviewing. A criterion-based sample cohort of surgical fellowship applicants describe the unique challenges of virtual interviewing

Full size image
Table 2 Thematic analysis
Full size table

Theme 1: perceived data deficiency

Applicants reported having limited data on the fellowship programs as a result of not interviewing in-person. Factors contributing to this theme were categorized into four domains: (a) applicant’s lack of subjective feel for the program, (b) a heavier reliance on available objective data and word of mouth, (c) difficulty distinguishing between different institutions, and (d) a lack of applicant-to-applicant interactions.

Loss of subjective feel for the program

Participants consistently noted that they could not thoroughly assess the atmosphere in the program as a result of not being there in person and inability to get an “on the ground feeling”. They noted that virtual interviews failed to convey the culture of the program, camaraderie between attendings and fellows, the appearance and ambience of the local environment, and, most importantly, the applicant’s personal fit. Participants described difficulty in gauging the attitudes of the faculty and getting a sense of how they interact with one another and with the trainees, especially when the remote interview day consisted mostly of one-on-one interactions between the applicant and a series of interviewers. This deficiency was especially evident for participants who had initially interviewed in person and then transitioned to the virtual format: “I think I got a better understanding of the vibe of the programs themselves [referring to in-person interviews]. And I guess some of that vibe is the non-verbal cues and interaction that you had with the interviewers or faculty members. Is this a ‘happy place’ or is this a ‘not so happy place’? Even though they all say: ‘Oh, we’re all happy here’, there are these kinds of other clues, non-verbal clues, that are lost in the virtual environment” (Participant 11).

To a lesser extent, some participants also reported a loss of subjective feel regarding the hospital and/or city. They recommended including opportunities to observe team interactions, such as multi-faculty conversations and conferences, to showcase the interactions within a program and convey some feeling about the program’s culture. In fact, participants described getting a “better idea of what the program is like” when programs made a conscious effort to show faculty interactions where attendings “just talk and joke around”, as opposed to “being ‘stone faced’ over zoom” or “siloed in the zoom setting”. Some participants also expressed satisfaction with virtual interview tours of the hospitals, namely the operating rooms, although these tours failed to completely compensate for the loss of subjective feel.

Heavier reliance on available objective data and word of mouth

Some applicants described the need to rely more heavily on objective data and individual experiences of those affiliated or familiar with the program when deciding on their final rank order. Examples of objective data included case logs with detailed case breakdown, research publications, and hospital location. Word of mouth included insights from various stakeholders such as current and previous fellows/applicants, personal mentors, or even the “traditional reputation” of a program and stereotypical impressions of the hospital or city. Applicants highlighted the importance of triangulating objective data and insights from stakeholders as a way of reaching a more informed decision to avoid selecting a program based on what some described as “faulty information”. Participants described maximizing access to individuals with knowledge about the program, preferably unbiased or independent sources. Many emphasized the importance of informal conversations with current fellows, even after the interview day. Some participants suggested that program directors make program data more readily available and allow for independent access to current and previous fellows to maximize their ability to reach an informed decision (Table 3).

Table 3 Participant recommendations to fellowship program directors that address the challenges of the virtual interview process
Full size table

Difficulty distinguishing between different institutions

Some participants reported that remote interviewing resulted in a less memorable experience, limiting their ability to differentiate between programs when the time approached to rank them. Participants recalled how programs blend together at a certain point and described the difficulty in seeing each program as an individual place. In contrast, in-person interviews involving multiple adjunct experiences (traveling to a new city, being in the facility, interacting with faculty) helped differentiate between programs when the interview season was over. Some participants recommended that future applicants keep notes after interviews to help with this issue (Table 4). "You need to take the time afterwards to really solidify those moments in your memory… you need to just write everything down, because the biggest issue for me was when the time came […] to rank the programs, a lot of them started to not really stand out as well as I would have liked” (Participant 4).

Table 4 Participant recommendations to future fellowship applicants to address the challenges of the virtual interview process
Full size table

Lack of applicant-to-applicant interactions

Another common domain was the reduced, or even absent, interactions between applicants. These applicant-to-applicant interactions allow for informal data sharing about programs along the interview trail. Co-applicants were felt to be helpful in sharing knowledge about fellowship programs at their local institutions or for providing personal experiences from interviewing at programs that their peers had yet to interview at. “They can tell me what they thought of the place, [and] that kind of sets the tone for me when I go into that interview” (Participant 9). Interactions with co-applicants were also seen as useful connections with future colleagues in their respective field. This domain was more frequently raised by applicants from non-academic institutions and applicants that did not have a co-resident who applied to the fellowship match.

Theme 2: superficial personal connections

Participant opinions differed regarding their ability to connect with interviewers virtually as opposed to in-person. Some participants felt that interviews in the virtual setting tended to be very formulaic, structured in a question-and-answer format, and that they lost “in between” informal conversations important for attaining in-depth connections. Participants emphasized the reduced opportunities for informal conversations, or “chitchat”, in the virtual setting, such as walking into an office and recognizing a personal cue (a sign of preferred hobbies, sports team, artwork or others) that leads to an informal conversation. These interactions were minimized in the virtual setting as a result of not only the formulaic interview structure, but also the use of “conservative [virtual] backgrounds”.

On the other hand, some participants felt that once they got used to the virtual interview format and made some adjustments, it did not drastically hamper their ability to connect with the interviewer. “After people got used to it, it was a little easier to get a personal connection. I mean, it’s still much better to do it in person, but overall, after a few of them, it was relatively comparable” (Participant 14).

Theme 3: magnification of non-professionalism

Some participants expressed how they viewed organization on virtual interview days as a direct reflection of the fellowship program as a whole. This feeling was magnified in the virtual interview process because of the perceived data deficiency described above and the limited interactions between the applicants and the programs. “The atmosphere contributed to places that I went to in person, and then to places that I didn’t go to, things like… one place sent us interview dates with different time zones and everything was mixed up, and then the fellow wasn’t available. I wasn’t interested. I probably ranked it, but I ranked it really low. It was a really big deal to me if people hadn’t been able to come up with an organized approach. That would be the closest I could come to atmosphere, I guess” (Participant 8). Applicants extrapolate features of the interview day to the fellowship training. As such, if the interview day is disorganized or has unprofessional elements, the applicant assumes that the quality of the training is disorganized/unprofessional. Examples of interviewers’ unprofessional behavior included: eating during the interview, constantly moving around the room/in the chair during the interview, abruptly stopping an interview due to unexpected clinical obligations, inability to prevent/manage avoidable technical difficulties, and poor time management.

Participants emphasized that program directors should treat the interview process as professional as if it were in person (Table 4). Some explained that programs who took the virtual interview process just as serious as the in-person operation stood out, as it implied that the program itself may be better organized.

Theme 4: logistical frustrations

A recurring theme was the logistical difficulties of virtual interviews: the interview structure, the setup, and the technology.

Rigidity of the virtual interview structure

Some participants felt limited by the virtual interview platform because of its rigid time frame. They experienced frustration at being abruptly taken out of the interview room when the allotted time expired. One participant mentioned that the virtual interview often resulted in “awkward endings” and suggested alerting applicants and interviewers a few minutes before completion of the interview to allow them to properly summarize the meeting. The rigid time frame also heightened the feeling of superficial personal connections, as described above.

Complicated dynamics for group interviewers

A frequent challenge cited by participants involved being interviewed by more than one interviewer. Participants indicated it was difficult to converse with multiple interviewers at once using the virtual technology, especially if all of the interviewers shared the same camera and/or microphone during the interview.

No standard interview platform

The majority of participants were annoyed and confused by having to deal with multiple different virtual interview platforms (Zoom® [Video Communications, Inc; San Jose, CA], Google Hangouts® [Google; Mountain View, CA], Microsoft Teams® [Microsoft Corporation; Redmond, WA], etc.). Fellowship programs used various software that required specific installations and adjustments by the applicants. This was especially challenging for the spring 2020 interview season, as widespread use of online video conference technology was still in its infancy.

Discussion

The COVID-19 pandemic forced programs to suspend in-person interviews within the United States, forcing surgical training applicants and programs to pivot to an online format [3,4,5,6]. Since the transition one year ago, programs and applicants have learned a lot about the virtual interview process. However, in order to prepare for the next virtual interview cycle, we present 2020–2021 surgical fellowship applicants’ perspectives on the challenges of and recommendations for virtual interviewing. Participants reported that a major disadvantage to the virtual interview process was a lack of knowledge about the fellowship programs. Filling in this information gap was deemed to be a burden and led to an increased dependence on published data and word of mouth when applicants finalized their fellowship rank lists. Interviewees felt that the virtual format magnified the non-professional behavior of interviewers and that poor organization of the virtual interview day prominently detracted from an applicant’s experience, likely because other in-person factors typically counterbalance those feelings.

These study results have important implications for future virtual interviews. Given the increased dependence on objective data, such as case breakdowns or descriptions of on-call responsibilities, applicants suggested that program directors make this information more readily available prior to the interview day and also offer informal independent interactions with current and previous fellows. These strategies may be especially helpful to smaller or newer surgical fellowship programs that are still developing their reputation and may lack broad networks of former trainees. Additional emphases should be on the organization and professionalism surrounding the interview day. Previous 2-day interview experiences that included a tour of the facility, communal dining, “happy hours”, and other opportunities for informal interactions have now been reduced to a few hours of virtual interactions. In this setting, the organization of the virtual interview day was often interpreted by the applicants as a microcosm of how the overall program was structured. Thus, as the “stakes are higher”, program directors seeking to attain higher rankings from favored applicants should attempt to replicate the formality and activities of the in-person experience as much as possible.

Our findings align with the personal perspectives and viewpoints published since the transition to virtual interviews for graduate medical education [3, 5, 11, 14]. One study noted the importance of maintaining professionalism in the virtual setting to improve the applicant’s perception of the program [4]. This paper also stressed the importance of providing applicants with collateral sources of information to compensate for being unable to visit the programs in person, data that will help them “vividly imagine themselves moving to and training in a new place”. These recommendations were confirmed in a quantitative analysis of fellowship applicants in the 2020 match that compared those interviewed on-site with those who interviewed virtually [15]. In this survey of 23 complex general surgical oncology fellowship applicants, 100% of those who interviewed in person felt they were able to get an “adequate understanding of the program’s culture”, while only 64% of those interviewed virtually felt that way. In a different survey of 17 Advanced GI/MI surgical fellowship candidates that were interviewed virtually during the 2020 cycle, negative experiences of the applicants were most commonly secondary to technical issues regarding the use of software [16].

Studies of applicant perspectives on virtual interviews for graduate medical education prior to COVID-19 give further context to our findings. In a survey of endocrine surgery fellowship applicants from 2013 to 2019, the most important goal of the interview was meeting the faculty, followed by “behind the scenes information” s[17]. In a 2018 study on the efficacy of videoconference interviews in the pediatric surgery match, most applicants disagreed or remained undecided when asked whether the virtual interview “allowed me to decide if the program was the right ‘fit’ for me” [18]. In a prospective randomized trial of applicants to a urology residency match at the University of New Mexico, researchers compared applicants randomized to a web-based interview to those interviewed on-site [8]. Interviewing virtually was associated with applicants feeling less comfortable ranking the program based on the interview, compared to in-person interviewing, although both groups declared they had a “good understanding” of the residency program. Interestingly, their set-up for virtual interviewing was as similar as possible to the traditional in-person process.

This study was conducted over 6 months after the 2020 fellowship interview season and after applicants received their match results, giving them ample time to reflect on the process and its results. While overall satisfaction with the virtual process was not our research question, it is notable that despite most participants expressing initial anxiety or frustration about the shift to a virtual format, almost all reported strong satisfaction with the interview process and the results of their match.

Limitations

Although a criterion-based purposive selection method was used, there were still underrepresented groups in the final participant cohort. We had a higher-than-average representation of applicants from academic centers and those interested in MIS or Bariatric fellowship programs. Applicants from the south and west of the country were also underrepresented. Furthermore, we acknowledge the possibility that negative experiences might have elicited applicants to participate in the study, especially with a small respondent number. There may also be some degree of recall bias at play, particularly if there was a sense that many of the programs blended together. In addition, these data reflect perceptions of the first cohort of a match that abruptly changed from on-site to virtual interviewing. We acknowledge that, throughout the COVID-19 pandemic, virtual technology has improved and applicants’ and programs’ ability to adapt and accept virtual communication has undoubtedly increased as well due to necessity of the times. For these reasons, some of the challenges and recommendations may not be transferrable to this year’s interview cohort(s).

Conclusions

Applicant perceptions of the virtual interview process for the surgical fellowship match suggest that transitioning from on-site to virtual interviewing resulted in a loss of access to traditional information streams. Virtual platform interactions also disproportionately influenced applicants’ final fellowship ranking decisions. Participants felt the virtual interviewing challenges could be mitigated with the provision of additional objective program metrics and triangulation of independent data. Future research should evaluate the satisfaction of the virtually matched applicants with their fellowship experience, and ideally compare against peers who had done the traditional interview.