Introduction

The continuing provision for formal medical education is one of the many challenges faced by educators and administrators during the coronavirus disease 2019 (COVID-19) pandemic [1]. Although this particular challenge was vocalized by educators and trainees in the spring of 2020 [2,3,4,5], the pandemic remains a major barrier to the successful training of medical providers [6]. Early recommendations by the Liaison Committee on Medical Education to suspend medical student involvement from patient-related activities compounded by ongoing virtual didactics were only a few contributors to the widespread implementation of self-directed graduate medical education [3]. Reductions in hospital presentations and outpatient neurology visits [7, 8] have also scaled back the clinical experience among trainees [9]. Furthermore, the cancellation (or limitation) of academic conferences has curtailed many opportunities for education, networking, and career development among medical providers. The advantages and disadvantages of a transition to a highly virtual learning environment (e.g., using Online Educational Resources, OERs) have been summarized previously [10, 11], but not all programs have been able to convert their curricula quickly and effectively in response to the pandemic. Established OERs in the form of online publications and video and audio podcasts have provided healthcare professionals [12] and educators [13] with the opportunity to incorporate existing resources into a developing digital syllabus. Anecdotal reports indicate a heightened use of widely available content via social media, virtual grand rounds, webinars, and other resources—even before the advent of the COVID-19 pandemic [14].

Originally a grassroots movement, the creation of medical OERs has blossomed over the last two decades, with numerous examples of multi-author websites, resources sponsored or supported by medical associations and journals, and highly networked learning environments populated by all levels of medical education learners [15]. In neurology, podcasting has played a major role in building OER popularity over the past 10 years [16, 17]. The first academically designed, neurology-themed podcast was launched in 2006 by The Lancet Neurology. This inaugural program was followed in rapid succession by podcasts from the American Academy of Neurology (2007), the Journal of the American Medical Association Neurology (2011), the Journal of Neurology, Neurosurgery, and Psychiatry (2013), the Neurocritical Care Society (2017), and the American Neurological Association (2019), among other academic neurology societies and independent groups.

The reach and utilization of neurology-themed podcasts, however, have never been formally investigated. As many educators have turned to (and trainees have grown reliant upon) digital resources during the COVID-19 pandemic, the ADVANCE (Advocates for Digital, Visual, Audio, and Networked Clinical Education) investigators sought to explore how podcasts have filled this gap in medical education during the early months of COVID-19. As a secondary analysis of these data, we explored which programs and episode features garnered the greatest attention in the form of unique download counts. Awareness of popularity trends and program features may help guide future educators when producing their own digital educational syllabi, or selecting which existing programs may be most useful to incorporate into an existing curriculum.

Methods

Definitions

For the purposes of this study, a “participant” is defined as any producer, author, or editor of digital content that was generated for the primary purpose of medical education. “Programs” refer to the podcast, with “episode” referring to each unique podcast audio publication. Access to published content was summated in the form of “downloads” (the number of user requests to download, livestream, or otherwise play a particular podcast episode).

Study design

Data will be made available to any qualified researcher upon reasonable request. We conducted a retrospective, observational study of neurology-specific OER developers as a subset of the more broad ADVANCE dataset. The main results of ADVANCE will be published separately. Participants who produced either (1) audio and/or video podcasts or (2) blogs for the purposes of medical education (April 01, 2019–May 31, 2020) were recruited based on affiliation with the ADVANCE principal investigators (JES, PB, and TC) for related studies on blogs and podcasts [16, 17], or were invited to participate based on affiliation with an academic society and/or peer-reviewed medical journal. Additionally, participation in the ADVANCE study was offered to any qualified producer, author, or editor by way of social media posting and word-of-mouth. Participants were not financially compensated for their contribution to the ADVANCE study. See Fig. 1 for inclusion flowchart.

Fig. 1
figure 1

Inclusion flowchart

Full size image

Neurology programs were included if they (1) reported a mean view/download count of greater than 2000 per month for their blog or podcast during 2019 (set as a minimum standard for notable impact for publications, as described previously [18]), (2) they were affiliated with an academic society or peer-reviewed medical journal, or (3) they offer continuing medical education (CME) credit to their users. These criteria were set as a surrogate for impact, with traffic counts (unique downloads) in line with previously published setpoints for academic advancement and impact [18]. Programs were excluded if the target audience did not include practicing medical professionals, students from the medical field, or medical graduate/postgraduate trainees. Programs were also excluded if they did not publish content prior to January 1, 2020, or were unable to provide data for the study period.

A brief survey instrument was circulated to all participants in order to determine eligibility for inclusion in ADVANCE. If participants confirmed they met inclusion criteria and lacked exclusion scenarios, they were invited to submit detailed data regarding consecutive episodes including the primary medical specialty of their program (e.g., internal medicine, neurology), target audience (practicing professionals, physician trainees, students, other), social media presence, affiliation with an academic society or peer-reviewed medical journal, unique podcast episode downloads, and provision of CME credit for use of their program. Participants were also asked to provide the title and topic of individual publications and whether or not the episode included COVID-19 content. Due to the popularity and versatility of the platform (e.g., typewritten and graphic content, embedding of links), Twitter accounts and followers were used to characterize social media affiliation and reach. In order to protect anonymity of each program, the number of Twitter followers for each program are reported categorically (< 1 k, 1–5 k, 5–10 k, 10–25 k, 25–50 k, > 50 k). All data were self-reported by participants, with Twitter data confirmed by the senior author (JES) as of December 20, 2020.

Statistical analysis

The primary outcome was the number of unique podcast episode downloads. Secondary outcomes included the number of downloads per unique program, unique downloads per episode, and the monthly downloads per program. Because users may have utilized several platforms in order to access content (e.g., personal computer, smart phone, tablet, or smart home device), it was not feasible to tabulate episode downloads for a given user. Therefore, episode downloads were aggregated for each unique publication across all users. One program published early content (in 2019) using a subscription-based OER platform, and later transitioned much of its 2020 content to a free open-access medical education (FOAM) platform. Cumulative data from the subscription-based and FOAM-based platforms were included in the formal analyses.

Descriptive statistics were used for groupwise comparisons. Categorical data were compared using chi square test, or Fisher’s exact test when contingency table cell counts were 5 or less. Normality of continuous data was assessed histographically and confirmed using the Shapiro–Wilk test. Downloads were summated over the 24-h period after release and for the 30-day period following their release, as these are the most commonly reported indices of accessed program content on commercially available Rich Site Summary feeds.

In order to optimize sensitivity for capturing a difference in downloads during the COVID-19 pandemic, we performed 2 comparisons. First, downloads were compared between all months using the Kruskal–Wallis equality of populations rank test, with Dunn’s pairwise correction using the Holm-Sidák method for multiple testing as previously described [19]. Second, downloads were compared between the immediate pre-COVID-19 months (January–March 2020 vs. April–May 2020) using the Wilcoxon signed-rank test, with pre-specified subgroup analyses of 30-day downloads stratified according to provision of CME credit, affiliation with academic society or peer-reviewed journal, episode format (interview with author, topical review, or other), and target audience (trainees vs. all other). The Pearson correlation coefficient was used to estimate the relationship between overall downloads, total episodes, and monthly episodes. A word cloud was generated using freely available online software in which the more frequently used words in episode titles corresponded with larger words in the titles (WordArt.com).

All analyses were conducted using STATA 15.0 (College Station, TX) on the two-sided level. p values are provided for convention with a significance level set at 0.05.

This study did not meet criteria for human subjects research; therefore, approval by the Institutional Review Board of the data coordinating center (Cooper University Hospital) was not required.

Results

Of the 43 surveyed producers, 12 were producers and/or editors of neurology-themed blogs and/or podcasts, 8 of whom completed the survey for this pre-specified subgroup analysis (66.7% response rate). Five programs met formal inclusion criteria (Fig. 1), and all were podcasts. Although social media posting and word-of-mouth were utilized to passively recruit additional participants, there were no other respondents recruited in this manner. Four programs had formal affiliations with an academic society and a peer-reviewed neurology journal (median 2019 impact factor of 8.55) and 1 program was independently published without formal affiliation with an academic society or peer-reviewed journal (Table 1).

Table 1 Program descriptions
Full size table

Monthly downloads

The median monthly download count was 2865 (IQR 869–7497) among podcasts, with significant differences in downloads between programs (p < 0.001; Fig. 2). While there was a 358% increase in total downloads during April 2020 when compared to the previous month (median 8124 [IQR 2913–14,177] vs. 2268 [IQR 540–6116], padj = 0.80; Fig. 2B), this did not achieve statistical significance in unadjusted groupwise comparisons (p = 0.45) or adjusted pairwise comparisons (padj > 0.05 for all pairwise comparisons). When comparing downloads from April 2020 versus April 2019 (median 1490, [IQR 238–2546]), there remained no statistically significant difference despite there being a 545% increase (padj = 0.27). All programs reported an increase in downloads in April 2020 versus the seasonal control month (April 2019) and versus the preceding month (March 2020). In May of 2020, there was a non-statistically significant, but visible normalization of download counts.

Fig. 2
figure 2

Download counts by program and month. A Monthly downloads (median with interquartile range) stratified by program. B Program downloads (median with interquartile range) stratified by month. Although there is a visible increase in monthly downloads (episode downloads), there were no significant differences appreciated between month-to-month comparisons. C Downloads within 24-h, 7-day, and 30-day periods following date of episode publication. Episodes are grouped by month according to date of publication (e.g., an episode that occurred on March 17, 2020, would be grouped in the March 2020 period). Although there was a small, visible increase in 30-day downloads during April 2020, there were no significant differences in downloads across 24-h, 7-day, or 30-day periods. D Unique episodes by month, stratified by presence or absence of COVID-19 content. *p ≤ 0.01; **p ≤ 0.001. All p-values shown here are adjusted using the Holm-Sidák method. COVID-19 denotes coronavirus disease 2019

Full size image

Episode characteristics

Fifty-two unique episodes were published during the study period. All episodes included content that was directed at practicing professionals (100%), while 88% of episodes included content intended for residents, fellows, and medical students, 58% intended for patients, and 58% also intended for non-medical listeners.

There was no significant difference in downloads per unique episode when compared across months in adjusted or unadjusted analyses (unadjusted p24h = 0.91, p7d = 0.57, p30d = 0.63; all adjusted pairwise p > 0.05; Fig. 2C). The non-significant increase in overall downloads during April 2020 corresponded to an increase in unique episodes during that month (r = 0.48, p = 0.003), but bore no relationship to the number of downloads per unique episode.

Subgroup analyses

Although there was no significant increase in 30-day download counts during the COVID-19 months versus the preceding months, one program (anonymized program no. 4) reported significantly greater downloads per unique episode. This was mediated by the program’s transition for some of its content from a subscription-based OER to a FOAM-based (Free Open Access Medicine) OER. By increasing the accessibility of its content to any user, the number of 30-day downloads increased by a mean of 2247 per unique episode that was made freely available (a mean growth of 375% per episode). The increase in download numbers attributed to this free OER medium (YouTube) showed continued growth over time as more episodes were published. The first episode accumulated 591 additional downloads over 30 days (99% gain) and the third episode accumulated 5280 additional downloads (605% gain).

CME credit was made available by 2 programs (31% of episodes), but 30-day downloads were 73% lower for episodes in which CME was offered (median download count 602 [IQR 323–762] vs. 2210 [IQR 1171–2845], p < 0.001). There was no difference in 30-day downloads during the months of COVID-19 based on availability of CME credit (Table 2).

Table 2 Subgroup analysis of 30-day downloads
Full size table

Four of the five included programs were affiliated with an academic society or peer-reviewed journal, which reported significantly lower 30-day downloads than the program which was not affiliated with an academic society or journal (median 734 [IQR 297–1881] vs. 2769 [IQR 2551–2888], p < 0.0001). There was a trend toward higher download counts during the COVID-19 months versus the preceding months among programs with an academic society affiliation (p = 0.06; Table 2), which was driven by anonymized program no. 4 for reasons outlined above.

The most frequently cited format for episodes was “topical review” (76%), followed by “interview with investigator/author” (11%) and other/unspecified (13%). There were non-significantly higher 30-day downloads among episodes designated as interviews (median 2210 [IQR 2157–2269]) when compared to reviews (1489 [IQR 405–2726]) and other episodes (428 [IQR 222–2943]; p = 0.31). There was no difference in 30-day downloads during the months of COVID-19 based on episode format (Table 2).

The majority of episodes included content that targeted medical trainees (88%), but there was no significant difference in 30-day downloads based on whether episodes targeted trainees or not (median 837 [IQR 374–2726] vs. 2210 [IQR 2157–2269], p = 0.18). Similarly, there was no significant difference in 30-day downloads during the months of COVID-19 when episodes were stratified according to target audience being trainees or non-trainees (Table 2).

While the most common, singular term included in episode titles was “COVID” (n = 6; 10 episodes featuring COVID-19 content; Fig. 3), there was no difference in 30-day downloads among episodes including COVID-19 content versus not (median 1979 [IQR 791–2873] vs. 1171 [IQR 405–2665], p = 0.28).

Fig. 3
figure 3

Word cloud generated from the most frequent text found in episode titles

Full size image

Discussion

In the wake of the global response to COVID-19, there was a slight, but non-significant increase in neurology podcast utilization as compared to previous months. The increase was also evident, although not statistically significantly different, when compared to the identical month in the preceding year. This increase in podcast utilization appears driven by the increase in the number of published episodes during April 2020 rather than the increase in unique episode downloads. Therefore, we cannot conclude that there was any significant increase in usage of podcast-based medical education during the COVID-19 pandemic. Unique episode download counts were unchanged during the early phase of the COVID-19 pandemic, and although there is no way to confirm this, it is most likely that users continued to access program content with similar frequency during the pandemic as in preceding months.

As a secondary objective to this analysis, we sought to determine which program or episode features were associated with the greatest number of unique downloads. When podcast episodes were organized according to the type of content (e.g., COVID-19-related or not, provision of CME credit, and format of episode), there was no significant difference indicating one episode type was more popular than any other. That said, there were numerically more downloads per podcast episode among episodes structured as “interview with author.” This may indicate that listeners prioritize investigator credibility, peer-review of published written material, and evidence-based medical practice. Many of these features have already been established as quality indicators in blogs and podcasts [20]. However, the fact that the program with highest monthly downloads was not affiliated with a peer-reviewed journal or academic society (and that it published content exclusively categorized as “topical review” or “other”) suggests that episodes featuring author interviews may not be the only route to a popular program. It is possible that, although the target audiences are likely similar across each program, the individual listeners may differ.

Of note, one podcast began to move some of its content from an entirely subscription-based OER to a FOAM-based OER. By offering content to users for free, the number of 30-day downloads increased by an average of more than 2200 per unique episode. The gains in download numbers showed continued growth as more episodes were published to YouTube, and should be expected to continue climbing as new users subscribe to this YouTube channel. At this time, this program does not plan to move all of its content to a FOAM-based OER medium, but is leveraging this platform to provide “general neurology” content to non-neurology users who might otherwise not have accessed their program’s content. Importantly, while content from this program was more accessible on YouTube, many institutional firewalls prohibit access to YouTube and other web pages. As podcasts are available on nearly any Wi-Fi or cellular device and can be streamed from a variety of freely available platforms, content that is distributed in podcast form will be more accessible within healthcare institutions.

Limitations

The single greatest limitation to this study is its small sample size and small number of overall neurology-themed podcasts. This limitation is expected to be rectified by the more comprehensive analysis of the ADVANCE dataset which is underway. Although the survey response rate among neurology-themed programs was high (67%), there are few neurology podcasts with generally infrequent episode publications and many episodes with small download counts by academic standards [18]. The small sample size raises the possibility of a type II error and precludes other exploratory analyses. However, many of these comparisons are being reported separately using the comprehensive ADVANCE dataset, which includes non-neurology-themed podcasts as well. The limited available data regarding podcast utilization, accessibility, duration, and user demographics also preclude certain potentially useful analyses. User location (e.g., country), user background (e.g., practicing professional, medical student, patient), and individual frequency of usage are not reported by standard podcast platforms. It is possible—although unlikely—that there was a shift in the type of users who accessed program content during the COVID-19 pandemic (e.g., fewer patients and more healthcare professionals), but we are unable to confirm this.

Conclusions

Despite the shift in medical education to the virtual classroom during the COVID-19 pandemic, there was no significant increase in neurology-themed podcast utilization early in the course of the pandemic. The COVID-19 pandemic may not have been a significant contributor to neuromedical podcast utilization, but we found that some podcast features (such as author interviews) may be more attractive among listeners, while other features (such as opportunities for CME credit) may not contribute significantly to podcast popularity. Ensuring widespread availability of podcast content using free platforms (such as iTunes and YouTube) can dramatically increase the utilization of these educational programs. We are actively exploring which program features and styles are most popular among medical listeners in a comprehensive analysis among neurology—and non-neurology-themed podcasts and blogs.