https://orcid.org/0000-0003-1180-1439
Figures
Abstract
Background
Social innovations in health are inclusive solutions to address the healthcare delivery gap that meet the needs of end users through a multi-stakeholder, community-engaged process. While social innovations for health have shown promise in closing the healthcare delivery gap, more research is needed to evaluate, scale up, and sustain social innovation. Research checklists can standardize and improve reporting of research findings, promote transparency, and increase replicability of study results and findings.
Methods and findings
The research checklist was developed through a 3-step community-engaged process, including a global open call for ideas, a scoping review, and a 3-round modified Delphi process. The call for entries solicited checklists and related items and was open between November 27, 2019 and February 1, 2020. In addition to the open call submissions and scoping review findings, a 17-item Social Innovation For Health Research (SIFHR) Checklist was developed based on the Template for Intervention Description and Replication (TIDieR) Checklist. The checklist was then refined during 3 rounds of Delphi surveys conducted between May and June 2020. The resulting checklist will facilitate more complete and transparent reporting, increase end-user engagement, and help assess social innovation projects. A limitation of the open call was requiring internet access, which likely discouraged participation of some subgroups.
Citation: Kpokiri EE, Chen E, Li J, Payne S, Shrestha P, Afsana K, et al. (2021) Social Innovation For Health Research: Development of the SIFHR Checklist. PLoS Med 18(9): e1003788. https://doi.org/10.1371/journal.pmed.1003788
Published: September 13, 2021
Copyright: © 2021 Kpokiri et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: The work received support from TDR, the Special Programme for Research and Training in Tropical Diseases co-sponsored by UNICEF, UNDP, the World Bank, and WHO (grants to PA and JDT). This project was also supported by grant NIAID K24AI143471 of the US National Institute of Allergy and Infectious Diseases (grant to JDT). The funders of the study had no role in the study design, data collection, data analysis, data interpretation, or writing of the report.
Competing interests: I have read the journal’s policy and the authors of this manuscript have the following competing interests: LGC is Senior Advisor on Research for Health at the Pan American Health Organization and has contributed to SIHI Global and the SIHI-LAC (Americas) coordination. Contributions to this article do not necessarily reflect the policies or positions of his employer.
Abbreviations: COVID-19, Coronavirus Disease 2019; LMIC, low- and middle-income country; PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses; SIFHR, Social Innovation For Health Research; SIHI, Social Innovation in Health Initiative; TIDieR, Template for Intervention Description and Replication; USSD, Unstructured Supplementary Service Data
Introduction
Social innovations in health are inclusive solutions to address the healthcare delivery gap that meet the needs of end users through a multi-stakeholder, community-engaged process [1]. Many social innovations have been developed in response to specific community needs. A subset of social innovations have transformed health service delivery in low- and middle-income countries (LMICs). For example, social innovations have expanded private sector pharmacy services to manage childhood illnesses in Uganda [1], improved housing, and addressed environmental risks, leading to reduced infestation rates for Chagas disease in Guatemala [2] and increased gonorrhea and chlamydia testing among sexual minorities in China [3]. While these social innovations have shown promise, research is needed to test, implement, adapt, and scale up innovations and their impact [1]. Social innovation in health may strengthen health systems and help to achieve the Sustainable Development Goals of the United Nations Agenda 2030 [1,4].
Research checklists provide one practical way to formalize and standardize the reporting of research findings. Research reporting standards have greatly developed in the past 2 decades, leading to dedicated clearinghouses and collaborations such as the EQUATOR Network, REWARD Campaign, and explicit advocacy and endorsement of such standards in intergovernmental policies and high-level documents aimed at increasing the value of research and reducing research waste [5–8]. Research checklists can spur multidisciplinary research [9,10], increase transparency [9,11,12], improve reporting completeness[9,11–13], and facilitate easier comparison and replicability of study results and findings [9,13,14]. While some checklists are focused on reporting methods [14] and others focus more on the details in reporting results [13], there are some checklists that report on both methods and results [11]. Overall, these checklists help researchers plan, execute, and report their processes and outcomes. However, to our knowledge, there has been only one research checklist that focuses on similar issues in global health [9]. In addition, meetings led by the Social Innovation in Health Initiative (SIHI), a network of international partners convened by TDR (the Special Programme for Research and Training in Tropical Diseases, cosponsored by UNICEF, UNDP, the World Bank, and WHO), highlight the need for research tools to advance social innovation in healthcare delivery in LMICs [15–18]. The purpose of this manuscript is to describe the development of a research checklist to assess and report social innovation projects as well as highlight the importance of research in social innovation projects.
Methods
Our working group used a 3-step process, including an open call for ideas, a scoping review, and a modified Delphi process (Fig 1). This 3-step process resulted in the development of a Social Innovation For Health Research (SIFHR) Checklist as well as a Social Innovation in Health Monitoring and Evaluation Framework [19].
M&E, monitoring and evaluation.
Open call
Social Entrepreneurship to Spur Health (SESH) is the research hub in China within the TDR SIHI. SESH and SIHI jointly organized a global crowdsourcing open call to solicit creative ideas and tools on the development of a social innovation research checklist, as well as ideas on measuring social innovation in health performance to develop a conceptual framework for measurement and evaluation. Crowdsourcing open calls invite individuals or groups to solve a problem together and then share the solutions with the public [20]. The purpose of the checklist was to develop a list of key components related to social innovation in health research. The measurement ideas were to help project managers and their teams effectively implement their social innovation projects, guide and improve project design, and allow them to more accurately report and measure the impact of their projects.
We formed a steering committee (JT, IW, BH, JF, PA, KM, KA, EK, UA, and SP) to finalize the call for submissions, decide the prize structure, identify judges, and advise on implementation. Steering committee members for this open call included researchers, innovators, policymakers, implementers, and students. This process was similar to other crowdsourcing open calls organized by SESH to understand research mentorship in LMICs [21] and to promote HIV testing and hepatitis testing where online open calls led to in-person consensus building meetings for further action [22,23].
The open call was launched on November 27, 2019 and closed on February 1, 2020. During this time, the open call was distributed within the SIHI network, through social media channels (e.g., Twitter), on SESH’s website, and through other partner and academic networks. The open call solicited monitoring and evaluation frameworks, research checklists, and methods for assessing monitoring and evaluation. Eligibility criteria included written in English, less than 1,000 words, and contained a document or attachment that provided a rationale and explanation for either a monitoring and evaluation framework or a research checklist. Volunteer judges were selected, with a focus on people in LMICs who have experience in social innovation. The focus on strong participation from LMICs was because social innovations are community engaged and locally driven. Too often in global health, high-income country researchers make key decisions that influence the process and outcomes. Our intention of ensuring strong LMIC participation within the steering committee and judging group was to increase the likelihood that this research checklist would be relevant to many people in LMICs. After the open call was closed, each submission was screened independently for eligibility, and eligible entries were reviewed by 5 independent judges. The open call can be found in S1 Text.
Entries were subjectively judged by participants in 3 categories: (1) relevance to inform a standardized framework or research checklist; (2) creativity; and (3) the participant’s experience in the field of social innovation. Scores were assigned between “1” and “10” in each category and then averaged for a final score of the entry. Entries that achieved a mean score of “7” and above were deemed semifinalists. Semifinalists entries were then reviewed once more by the steering committee, and finalists were selected. Finalist submissions were chosen by the steering committee in March 2020 and invited to join a hackathon to finalize the research checklist. Hackathons are a form of crowdsourcing that include an open call for participants, a sprint collaborative event, and follow-up activities [24].
Given the Coronavirus Disease 2019 (COVID-19) pandemic, we transitioned our originally planned in-person workshop to a digital consensus building process following the open call. We asked participants to choose between a day-long videoconference versus shorter sessions over 2 to 3 days and most wanted the latter. We organized an intensive period of collaboration via videoconference calls over the span of several weeks plus 2 separate 2-hour videoconference workshops. We scheduled an additional videoconference focused on introductions and logistics. Further details about the hackathon’s digital consensus building process are described in the section on the modified Delphi process below.
Scoping review
The steering committee reviewed peer-reviewed literature and gray literature related to social innovation in health to understand the current landscape and existing research and practice efforts in this field. We organized a scoping review based on the framework outlined by Arksey and O’Malley and used the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Checklist for scoping reviews. The purpose of a scoping review is to examine the depth, breadth, and nature of research on a given topic. This scoping review focused on the monitoring and evaluation related to social innovation in health. We searched PubMed, Cochrane, ClinicalTrials.gov, the EQUATOR Network database, Google Scholar, and Scopus. Included publications examined social innovation in health from the perspective of monitoring and evaluation, with a particular focus on informing a conceptual framework and research checklist. This included conceptual frameworks, research checklists, empirical monitoring studies, evaluation approaches, and similar types of articles. The search strategy included variations of the following terms: social innovation, monitoring, evaluation, conceptual framework, and research checklist. We exported records to EndNote.
Inclusion criteria were conceptual frameworks or research checklists focused on or related to social innovation in health. Studies related to social innovation that mentioned health, but did not focus on health, were also included. We included checklists relevant to the monitoring and evaluation of social innovation projects and research. We excluded records with minimal relevance to social innovation, those only related to programs, systematic, and scoping reviews, and manuscripts not written in English.
JT and EK independently reviewed titles and abstracts for inclusion. Discrepancies were brought to the SIHI working group on monitoring and evaluation. Full-text articles of relevant manuscripts were shared with the entire working group using file sharing software. The initial search was performed on May 5, 2020 and updated on November 30, 2020.
Checklist development
We followed recommendations from the Guideline International Network [25] and completed a checklist relevant to guideline development (S2 Text) [26].
Modified Delphi process
The Delphi process is a structured method to develop consensus and is commonly used to develop health guidelines and research checklists [26]. A typical Delphi process has a group of experts iteratively develop a consensus. Given the importance of end users in social innovation, our Delphi process was modified to incorporate feedback from expert (all 3 rounds) and end users (rounds 1 and 2). The expert group consisted of the steering committee and finalists from the crowdsourcing open call. The user group included people with experience and/or interest in social innovation research. Iterative feedback from each of the 3 Delphi surveys (S3 Text) was used to revise the research checklist and the monitoring and evaluation conceptual framework. In the first round, comments suggested the need for more details on the methodology, including examples and key definitions (S4 Text). In round 2, feedback highlighted the need for social innovation impacts, strengths, limitations, and end-user perspectives. The third and final round refined the format of the research checklist. Our team reviewed and analyzed the results of the online surveys between each round of voting.
Results
Open call
We received a total of 21 unique submissions from 12 different countries: United States of America (n = 5), Bangladesh (n = 3), Colombia (n = 2), Nigeria (n = 2), Philippines (n = 2), Cameroon (n = 1), Guinea (n = 1), Honduras (n = 1), India (n = 1), Kenya (n = 1), Thailand (n = 1), and United Kingdom (n = 1). Therefore 65% (11 out of 17) of the unique submissions (all those except entries submitted from the USA and the UK) were from LMICs. After the initial screening, 17 out of the 21 submissions were deemed eligible for judging. After the steering committee discussion, 4 finalists were selected: 2 from the USA, 1 from the Philippines, and 1 from Bangladesh.
We noted several themes across entries received, including the following: a strong focus on community and stakeholder engagement; considering implementation as an essential component; and examining financial models and financial sustainability. The eligible entries provided initial frameworks to examine social innovation in health projects at different stages and suggested processes and data that should be reported to enable evaluation of project effectiveness and impact.
Discussions at workshops
Two 2-hour online workshops were organized on May 19 and May 29 of 2020. The videoconference workshops provided an opportunity to discuss the research checklist based on data from the open call and the findings of the scoping review. Participants included steering committee members and open call finalists. For example, one of the major topics of discussion at our second meeting focused on the topic of financing and how sustainability and revenue generation activities are not consistently reported. The discussion uncovered that some participants felt that financing and sustainability should be explicitly included in the research checklist. We included this item in the draft research checklist and used the modified Delphi process to determine the content of the final version of the checklist.
Delphi surveys
In between workshops, social innovation experts, end users, and the broader SIHI network were asked to complete online surveys as part of the modified Delphi process. Experts were involved in all 3 Delphi surveys. End users had a separate group to review and were involved in the first 2 Delphi surveys. The broader SIHI network was consulted in the first survey only. The first Delphi survey was completed by 65 out of 96 (68%) invited participants during May 1 to May 5, 2020. Of these, we had 18 males and 47 females from 20 different countries. There were 18 participants from high-income countries and 47 from LMICs across Africa, Asia, and Latin America. More than half of respondents (65%) had previously done research in social innovation in health. Overall responses included structuring the preamble with mission statement and adding important definitions, specifying and clarifying each checklist item, and defining terms used such as health, stakeholders, facilitators versus providers, and open-access resources. Feedback during the first few consensus building videoconference meetings was further incorporated such as including additional items, limitations, and strengths.
The second Delphi survey was conducted from May 29 to June 2. It was completed by 22 out of 45 (49%) invited participants. An end-user meeting was also convened to solicit innovators perspective into the research checklist elements as a separate digital meeting. Based on feedback, we added more specific descriptions of social innovation, ensured consistency across key terms, and provided illustrative examples.
The final Delphi survey was completed by 16 out of 25 (64%) invited participants. Minor adjustments at this stage included fixing grammatical errors and harmonizing definitions. Some themes and specific feedback received from each round are provided in S4 Text.
SIFHR Checklist.
Our social innovation in health research checklist uses a variety of terms that are defined differently across disciplines. The social innovation research checklist is adapted from the Template for Intervention Description and Replication (TIDieR) Checklist that focuses on better reporting of interventions.[13] Key terms are defined in Table 1, and we use WHO definition of social innovation.
At the end of our multistep process, we finalized a research checklist with 17 items (Table 2, S5 Text). Table 2 includes the social innovation in health research checklist, a description of each of the items, and the percentage of Delphi survey respondents who affirmed that each item should be included in our final survey. To determine interrater reliability, we computed a Cohen kappa coefficient (Cohen’s K = 0.7). We have also included a Supporting information file with the checklist in PDF format along with a list of useful resources and additional information about the SIHI research hubs. We gathered this set of resources from steering committee members and finalists during our checklist development process. In addition, we list 3 examples of a completed checklist in Table 3. They describe social innovation research on Chagas disease in Guatemala [2], maternal health in Uganda [23], and sexual health in China [3].
Discussion
The SIFHR Checklist will help to democratize research in social innovation in health and enhance the rigor of research on social innovation in health. It is intended for research on social innovation in diverse global settings, especially in LMICs. The research checklist will help to structure research studies, standardize research reporting, and provide guidance for routine monitoring and evaluation related to social innovation in health. Our research checklist extends the literature by focusing on social innovation in health, including iterative feedback from end users at multiple steps, and using inclusive crowdsourcing methods.
Our crowdsourcing open call and digital hackathon provided new methods for inclusive end-user feedback, including end users in LMICs. The process of consensus development is typically driven by experts, and many argue that including other stakeholders is essential [25]. Crowdsourcing open call methods have been used in other health research projects to aggregate wisdom from diverse groups of people [27]. The process involved end users at all stages of the project, including the modified Delphi process that finalized the checklist. Given the recognized importance of end users in health [28], our process for consensus development may be relevant to other guideline development at the national or global level.
Our digital hackathon provided an opportunity to transition an in-person method to a series of online workshops. Most hackathons to date have focused on intense in-person collaboration [24]. Potential benefits of the digital hackathon approach include broader inclusion of individuals who would not have been able to join an in-person event, increased time between events to process information and do additional research, and increased capacity to allow real-time participation from people across multiple time zones.
Our research checklist hackathon process has several limitations. First, the standards on research reporting of social innovation are still emerging. Second, the open call required internet access and was likely easier to access among English speaking academic researchers and those with high socioeconomic status; alternative methods to solicit ideas and contributions (e.g., Unstructured Supplementary Service Data [USSD]) could potentially broaden the reach of future open calls and increase contributions from individuals of lower socioeconomic status. Third, we only accepted submissions in English. However, previous global crowdsourcing open calls suggest that when all 6 official languages of WHO are options for submissions, greater than 90% are in English [29].
This research checklist has implications for research and policy. From a research perspective, this checklist will help people in diverse settings to design, implement, and disseminate social innovation in health research. Further research is needed to understand how to measure social innovation in health. Our research checklist raises questions about optimal methods for designing, implementing, and disseminating social innovation in health research. From a policy perspective, our digital hackathon provides an efficient method for collaborative consensus development that is well suited to the COVID-19 era. This could be relevant to policymakers and health leaders organizing consensus processes.
Conclusions
This 17-item social innovation in health research checklist expands the social innovation literature and will be iteratively improved. The SIFHR Checklist can lead to better health and social outcomes through more complete and transparent reporting of the development, implementation, and evaluation of social innovations in health. SIFHR can be used before, during, and after cocreating social innovations in health. Use of the research checklist will help to increase end-user and stakeholder engagement, increase the rigor of monitoring and evaluation strategies, consider plans for sustainability, and better determine social and health impacts of social innovation. We hope that researchers, innovators, and partners are able to learn more about the processes and results of social innovation for health research projects from each other and that this will drive improved social and health outcomes.
Supporting information
S1 Text. Social Innovation in Health Monitoring and Evaluation open call.
https://doi.org/10.1371/journal.pmed.1003788.s001
(DOCX)
S2 Text. Guideline International Network checklist for guideline development.
https://doi.org/10.1371/journal.pmed.1003788.s002
(DOCX)
S4 Text. Feedback obtained from Delphi surveys.
https://doi.org/10.1371/journal.pmed.1003788.s004
(DOCX)
S5 Text. SIFHR Checklist.
SIFHR, Social Innovation For Health Research.
https://doi.org/10.1371/journal.pmed.1003788.s005
(PDF)
Acknowledgments
We would like to thank all who contributed to the open call and participated in the Delphi surveys. We also thank Larry Han for his helpful feedback on an earlier version of this manuscript.
Disclaimers
The views expressed in this article are those of the authors and do not reflect the official policy or position of any organizations.
License information
Pan American Health Organization (PAHO), Regional Office for the Americas of the World Health Organization. This is an open-access article distributed under the terms of the Creative Commons Attribution IGO License (http://creativecommons.org/licenses/by/3.0/igo/legalcode), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. In any reproduction of this article, there should not be any suggestion that PAHO or this article endorse any specific organization or products. The use of the PAHO logo is not permitted. This notice should be preserved along with the article’s original URL.
References
- 1. Halpaap BM, Tucker JD, Mathanga D, Juban N, Awor P, Saravia NG, et al. Social innovation in global health: sparking location action. Lancet Glob Health. 2020;8(5):e633–e4. pmid:32353305
- 2. Castro-Arroyave D, Monroy MC, Irurita MI. Integrated vector control of Chagas disease in Guatemala: a case of social innovation in health. Infect Dis Poverty. 2020;9(1):1–9.
- 3. Yang F, Zhang TP, Tang W, Ong JJ, Alexander M, Forastiere L, et al. Pay-it-forward gonorrhoea and chlamydia testing among men who have sex with men in China: a randomised controlled trial. Lancet Infect Dis. 2020;20(8):976–82. pmid:32530426
- 4. Mothe J, Vacaflor LE, Castro-Arroyave DM, Cuervo LG, Saravia NG. Exploring social innovation in health in Central America and the Caribbean. Rev Panam Salud Publica. 2020;44. pmid:32774348
- 5. Altman DG, Simera I. A history of the evolution of guidelines for reporting medical research: the long road to the EQUATOR Network. J R Soc Med. 2016;109(2):67–77. pmid:26880653
- 6. Glasziou P, Altman DG, Bossuyt P, Boutron I, Clarke M, Julious S, et al. Reducing waste from incomplete or unusable reports of biomedical research. Lancet. 2014;383(9913):267–76. pmid:24411647
- 7.
World Health Organization. WHO’s Role and Responsibilities in Health Research: Draft WHO Strategy on Research for Health. Sixty-Third World Health Assembly. WHO: 2010.
- 8.
Pan American Health Organization. Pan American Health Organization 49th Directing Council 61st Session of the Regional Committee of WHO for the Americas. Washington, DC: 2009. Epub 2020 Dec 11. https://iris.paho.org/handle/10665.2/4792
- 9. Bazzano AN, Yan SD, Martin J, Mulhern E, Brown E, LaFond A, et al. Improving the reporting of health research involving design: a proposed guideline. BMJ Glob Health. 2020;5(2):e002248. pmid:32133198
- 10. O’Brien BC, Harris IB, Beckman TJ, Reed DA, Cook DA. Standards for reporting qualitative research: a synthesis of recommendations. Acad Med. 2014;89(9):1245–51. pmid:24979285
- 11. Tong A, Synnot A, Crowe S, Hill S, Matus A, Scholes-Robertson N, et al. Reporting guideline for priority setting of health research (REPRISE). BMC Med Res Methodol. 2019;19(1):1–11.
- 12. Duncan E, O’Cathain A, Rousseau N, Croot L, Sworn K, Turner KM, et al. Guidance for reporting intervention development studies in health research (GUIDED): an evidence-based consensus study. BMJ Open. 2020;10(4):e033516. pmid:32273313
- 13. Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ. 2014;348. pmid:24609605
- 14. Seale C, Silverman D. Ensuring rigour in qualitative research. Eur J Public Health. 1997;7(4):379–84.
- 15.
Special Programme for Research and Training in Tropical Diseases. Funding agencies explore options to advance social innovation in health. 2017.
- 16.
Special Programme for Research and Training in Tropical Diseases. Workshop Report—Co-creating a Research Guidance for Social Innovation in Health. 2018.
- 17.
Special Programme for Research and Training in Tropical Diseases. Social Innovation in Health Initiative Partner and Stakeholder Workshop. 2019.
- 18.
Special Programme for Research and Training in Tropical Diseases. Social Innovation in Health Monitoring and Evaluation Framework. Geneva, Switzerland: 2020.
- 19.
TDR/SESH/SIHI. Social innovation in health: monitoring and evaluation framework. 2021. https://socialinnovationinhealth.org/monitoring-evaluation-framework/
- 20.
TDR/SESH/SIHI. Crowdsourcing in health and health research: a practical guide. World Health Organization: 2018.
- 21. Bao H, Salisbury K, Chi BH, Tucker JD, Tang W. Research Mentorship Crowdsourcing Contest: Toward Global Health Good. Am J Trop Med Hyg. 2020;103(3):1270–3. pmid:32700667
- 22. Tang W, Wei C, Cao B, Wu D, Li KT, Lu H, et al. Crowdsourcing to expand HIV testing among men who have sex with men in China: a closed cohort stepped wedge cluster randomized controlled trial. PLoS Med. 2018;15(8):e1002645. pmid:30153265
- 23. Awor P, Nabiryo M, Manderson L. Innovations in maternal and child health: case studies from Uganda. Infect Dis Poverty. 2020;9(1):1–8.
- 24. Tucker JD, Tang W, Li H, Liu C, Fu R, Tang S, et al. Crowdsourcing designathon: a new model for multisectoral collaboration. BMJ Innov. 2018;4(2).
- 25. Qaseem A, Forland F, Macbeth F, Ollenschläger G, Phillips S, van der Wees P. Guidelines International Network: toward international standards for clinical practice guidelines. Ann Intern Med. 2012;156(7):525–31. pmid:22473437
- 26. Schünemann HJ, Wiercioch W, Etxeandia I, Falavigna M, Santesso N, Mustafa R, et al. Guidelines 2.0: systematic development of a comprehensive checklist for a successful guideline enterprise. CMAJ. 2014;186(3):E123–E42. pmid:24344144
- 27. Thangaratinam S, Redman C. The Delphi technique. Obstet Gynaecol. 2005;7:120–5.
- 28. Petkovic J, Riddle A, Akl EA, Khabsa J, Lytvyn L, Atwere P, et al. Protocol for the development of guidance for stakeholder engagement in health and healthcare guideline development and implementation. Syst Rev. 2020;9(1):1–11.
- 29. Tucker JD, Meyers K, Best J, Kaplan K, Pendse R, Fenton KA, et al. The HepTestContest: a global innovation contest to identify approaches to hepatitis B and C testing. BMC Infect Dis. 2017;17(1):177–85.