‘More of an art than a science’? The development, design and mechanics of the Delphi Technique

doi:10.1016/j.sapharm.2021.06.027

Research in Social and Administrative Pharmacy

Volume 18, Issue 1, January 2022, Pages 2230-2236

https://doi.org/10.1016/j.sapharm.2021.06.027 Get rights and content

Abstract

The Delphi Technique is a group judgement method which is typically used to reach agreement from a group of people with expertise in a particular area. It is an iterative process where panel members complete questionnaires over several rounds, often rating their agreement/disagreement against a statement, with changes made in later rounds based on the feedback received. It has been used widely in pharmacy-related studies relevant to education, research and practice. This paper provides a critical analysis of the various design choices which researchers may consider when planning a Delphi namely the panel of participants, the use of the Likert scale, the effect of feedback, what constitutes consensus and the number of rounds. It also gives an overview of the development and origins of the Delphi, and discusses the advantages and disadvantages of the technique. Advantages include that the Delphi can be conducted with panel members in different geographical locations in their own time, however the technique can therefore take longer to conduct and lacks face-to-face discussion. Patient experts may be less comfortable participating in a relatively complex survey, however the anonymous nature of the process can be more inclusive in allowing participants to feedback candidly. This paper shows the importance of careful planning of the design choices to ensure the reliability and validity of the Delphi.

Section snippets

Author statement

Sarah Drumm: Conceptualization, Methodology, Writing- Original draft preparation, Investigation, Project administration. Catriona Bradley: Supervision, Writing- Reviewing and Editing. Frank Moriarty: Methodology, Supervision, Writing- Reviewing and Editing.

The panel

Typically, the panel chosen to complete a Delphi comprises those who have experience or expertise in a particular field - sometimes referred to as experts or specialists, or stakeholders who form a representative response group and who are affected by, or who work in the area.⁸^,²⁸ Based on those selected, a certain level of knowledge about the field is presupposed and one would imagine that these would be the most appropriate respondents in a study. This may not always be the case however as

Origins of the Delphi

In July 1962 a memorandum was produced for the Research and Development, or Rand, Corporation, by Norman Dalkey and Olaf Helmer entitled An Experimental Application of the Delphi Method to the Use of Experts.³⁰ The report was a revised version of a classified report, The Use of Experts for the Estimation of Bombing Requirements (1951), with the aim of making the report suitable for a wider audience.⁶³ The original report remains classified so this abridged version is still used today.⁶⁴ The

Advantages and disadvantages

Advantages to using the Delphi include that it can be completed by respondents in different geographical locations in their own time which allows for reflection, and it is cost effective.⁸^,⁵³ The structured method encourages participation from all group members while avoiding the issue of heterogeneity or of a dominant member or manipulation, as can sometimes be seen with focus groups.²^,⁸^,¹⁴^,⁵¹ Participants in Delphis have commented that it is inclusive, comprehensive, rigorous, systematic and

Conclusion

The nebulous beginnings of the Delphi and the fact that the original report is still classified lends a certain air of intrigue to its development. However since this, the method has been extensively used and refined, with some degree of variation in aspects of its design. This suggests it is widely applicable, and flexible for use for a variety of purposes. The Delphi's geographical neutrality and web-based format has advantages over methods that require face-to-face meetings. Certainly with

Declaration of competing interest

None.

References (70)

B. Graham et al.
Delphi as a method to establish consensus for diagnostic criteria
J Clin Epidemiol
(2003)
S. Di Zio et al.
Real time spatial delphi: fast convergence of experts' opinions on the territory
Technol Forecast Soc Change
(2017)
U.G. Gupta et al.
Theory and applications of the Delphi technique: a bibliography (1975-1994)
Technol Forecast Soc Change
(1996)
G. Rowe et al.
The Delphi technique as a forecasting tool: issues and analysis
Int J Forecast
(1999)
J. Landeta
Current validity of the Delphi method in social sciences
Technol Forecast Soc Change
(2006)
P. Ecken et al.
Desirability bias in foresight: consequences for decision quality based on Delphi results
Technol Forecast Soc Change
(2011)
H.A. von der Gracht
Consensus measurement in Delphi studies. Review and implications for future quality assurance
Technol Forecast Soc Change
(2012)
G. Rowe et al.
Judgment change during Delphi-like procedures: the role of majority influence, expertise, and confidence
Technol Forecast Soc Change
(2005)
I.R. Diamond et al.
Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies
J Clin Epidemiol
(2014)
J.R. Covvey et al.
Use of a modified delphi process to determine course objectives for a model global health course in a pharmacy curriculum
Am J Pharmaceut Educ
(2018)

J. Penm et al.

Defining and identifying concepts of medication reconciliation: an international pharmacy perspective

Res Soc Adm Pharm

(2019)

A.K. Gibbins et al.

Managing inappropriate use of non-prescription combination analgesics containing codeine: a modified Delphi study

Res Soc Adm Pharm

(2017)

T. Jebara et al.

A modified-Delphi study of a framework to support the potential implementation of pharmacist prescribing

Res Soc Adm Pharm

(2020)

E.G. Trevelyan et al.

Delphi methodology in health research: how to do it?

Eur J Integr Med

(2015)

B. Weijters et al.

The effect of rating scale format on response styles: the number of response categories and response category labels

Int J Res Market

(2010)

G. Rowe et al.

Delphi: a reevaluation of research and theory

Technol Forecast Soc Change

(1991)

C. Okoli et al.

The Delphi method as a research tool: an example, design considerations and applications

Inf Manag

(2004)

J.V. Meijering et al.

The effect of controlled opinion feedback on Delphi features: mixed messages from a real-world Delphi experiment

Technol Forecast Soc Change

(2016)

F. Woudenberg

An evaluation of Delphi

Technol Forecast Soc Change

(1991)

J.A. de Boer

The Oracle at Delphi: the pythia and the pneuma, intoxicating gas finds, and hypotheses

A. Marchais-Roubelat et al.

The delphi method as a ritual: inquiring the delphic oracle

Technol Forecast Soc Change

(2011)

K. Fitch et al.

RAND/UCLA Appropriateness Method User's Manual

(2001)

S.S. McMillan et al.

How to use the nominal group and Delphi techniques

Int J Clin Pharm

(2016)

D. O'Mahony et al.

STOPP/START criteria for potentially inappropriate prescribing in older people: version 2

Age Ageing

(2015)

K.K. Janke et al.

A modified Delphi process to define competencies for assessment leads supporting a doctor of pharmacy program

Am J Pharmaceut Educ

(2016)

C. Pope et al.

Qualitative Research: reaching the parts other methods cannot reach: an introduction to qualitative methods in health and health services research

BMJ

(1995)

H.A. Linstone et al.

The Delphi Method - Techniques and Applications

(1975)

E.R. Holmes et al.

The impact of the internet on community pharmacy practice: a comparison of a delphi panel's forecast with emerging trends

Health Market Q

(2002)

F. Hasson et al.

Research guidelines for the Delphi survey technique

J Adv Nurs

(2000)

S. Hanafin et al.

The Delphi Technique: A Methodology to Support the Development of a National Set of Child Well-Being Indicators

(2004)

J. Jones et al.

Qualitative Research: consensus methods for medical and health services research

BMJ

(1995)

S. Drumm et al.

The development of an accreditation framework for continuing education activities for pharmacists

Pharmacy

(2020)

K.E. Watson et al.

Defining pharmacists' roles in disasters: a Delphi study

PloS One

(2019)

A. Rankin et al.

Core outcome set for trials aimed at improving the appropriateness of polypharmacy in older people in primary care

J Am Geriatr Soc

(2018)

E. Barry et al.

PIPc study: development of indicators of potentially inappropriate prescribing in children (PIPc) in primary care using a modified Delphi technique

BMJ Open

(2016)

Cited by (47)

Categorizing digital data collection and intervention tools in health and wellbeing living lab settings: A modified Delphi study
2024, International Journal of Medical Informatics
Health and Wellbeing Living Labs are a valuable research infrastructure for exploring innovative solutions to tackle complex healthcare challenges and promote overall wellbeing. A knowledge gap exists in categorizing and understanding the types of ICT tools and technical devices employed by Living Labs.
Define a comprehensive taxonomy that effectively categorizes and organizes the digital data collection and intervention tools employed in Health and Wellbeing Living Lab research studies.
A modified consensus-seeking Delphi study was conducted, starting with a pre-study involving a survey and semistructured interviews (N=30) to gather information on existing equipment. The follow-up three Delphi rounds with a panel of living lab experts (R1 N=18, R2 - 3 N=15) from 10 different countries focused on achieving consensus on the category definitions, ease of reading, and included subitems for each category. Due to the controversial results in the 2nd round of qualitative feedback, an online workshop was organized to clarify the contradictory issues.
The resulting taxonomy included 52 subitems, which were divided into three levels as follows: The first level consists of 'devices for data monitoring and collection' and 'technologies for intervention.' At the second level, the 'data monitoring and collection' category is further divided into 'environmental' and 'human' monitoring. The latter includes the following third-level categories: 'biometrics,' 'activity and behavioral monitoring,' 'cognitive ability and mental processes,' 'electrical biosignals and physiological monitoring measures,' '(primary) vital signs,' and 'body size and composition.' At the second level, 'technologies for intervention' consists of 'assistive technology,' 'extended reality – XR (VR & AR),' and 'serious games' categories.
A common language and standardized terminology are established to enable effective communication with living labs and their customers. The taxonomy opens a roadmap for further studies to map related devices based on their functionality, features, target populations, and intended outcomes, fostering collaboration and enhancing data capture and exploitation.
Pharmacy practice and social pharmacy forging ahead
2024, Research in Social and Administrative Pharmacy
Development of an adaptation framework to implement a new professional pharmacy service (PPS) to a new environment
2024, Research in Social and Administrative Pharmacy
The effective provision of professional pharmacy services is critical to support the delivery of primary health care. Structured frameworks and theoretical strategies are required to facilitate successful service implementation processes, outcomes and sustainability. This commentary discusses the considerations of what framework (adoption versus adaptation) would be suitable when implementing a new professional pharmacy service to a new environment. Utilizing Minor Ailments Services (MASs) as an exemplar as a professional pharmacy service case study, the research that underpinned these considerations enabled the development of a sequential, phased framework. There is the potential to utilize this framework for future evolving professional pharmacy services in the new setting.
Using the Delphi method for selecting energy efficiency indicators in the Thailand context
2023, Energy Reports
The restaurant buildings have consumed less energy than other commercial buildings, directly affecting their operation cost. One approach that can reduce energy consumption in restaurants is to have criteria for evaluating energy use, but there is no such criterion in Thailand. Therefore, to achieve the criteria in the guidelines, a detailed comparison of the overseas assessment tools on the similarities and differences will be helpful to achieve the new assessment tool on restaurant buildings for Thailand by using three tools from different countries, including (1) Green Restaurant 4.0 Standards, (2) Energy Star for Restaurants, and (3) BCA Green Mark for Restaurants. The result from this study adapted the modified Delphi method to determine the consensus by ten experts on the indicators. The result shows the five indicators: (1) install roof insulation, (2) air-condition room control, (3) encourage the use of consumption for the first time, (4) duct sealing in the building, and (5) encourage to reuse of furniture had passing expert’s judgment to be used as reference. The result will help develop the appropriate assessment tool for reducing overall energy consumption in Thailand.
Driving sustainable healthcare service management in the hospital sector
2023, Journal of Cleaner Production
Sustainability in service management is a crucial issue in today's competitive business environment. Previous researchers have conducted studies on healthcare sustainability from various areas, but none has been conducted on the assessment of driving factors to sustainable service management in the hospital sector. This study fills this research gap by identifying and assessing the driving factors behind sustainable service management in the hospital sector from an emerging economy context. The study does so by utilizing integrated qualitative and quantitative research methods. Primarily, the study performed a Delphi study to identify the most relevant driving factors and then utilized the integrated grey-based decision-making trial and evaluation laboratory (DEMATEL) model to aid the evaluation within the Bangladesh hospital sector. The study identified eleven factors as the causal and nine factors as effect group factors. Furthermore, the factor “Engaging experienced doctors in top management (D20)” was identified as the most prominent driving factor for sustainable hospital service management. The driving factor “Implementation of public health policies (D18)” was identified as the topmost causal factor, and “Proper maintenance and monitoring of hospital support services (D9)” identified as the most influenced/impacted effect group driving factor. The study's findings have significant policy implications for service operations and managers in the hospital sectors. The service managers should give special attention to the causal group driving factors and invest heavily in them as that will help to improve the effect group factors within the system.
A Delphi Study of Core Patient-Reported Outcomes for Advanced Renal Cell Carcinoma and Advanced Hepatocellular Carcinoma
2023, Seminars in Oncology Nursing
There is little research to help health care professionals understand what patient outcomes are considered a priority in advanced liver or kidney cancer. Knowing what is important to patients can help promote person-centered approaches to treatment and disease management. The aim of this study was to identify those patient-reported outcomes (PROs) that patients, carers, and health care professionals consider as “core” when providing care to those with advanced liver or kidney cancer.
A three-round Delphi study was undertaken to ask experts by profession or experience to rank PROs identified from a previous literature review. Fifty-four experts, including people living with advanced liver or kidney cancer (44.4%), family members and caregivers (9.3%), and health care professionals (46.8%), reached consensus on 49 PROs including 12 new items (eg, palpitations, hopefulness, or social isolation). Items with the highest rate of consensus included quality of life, pain, mental health, and capacity to do daily activities.
People living with advanced liver or kidney cancer experience complex health care needs. Some important outcomes were not actually captured in practice in this population and were suggested as part of this study. There are discrepancies between the views of health care professionals, patients, and family in what is important, highlighting the need of using measures to facilitate communication.
Identification of priority PROs reported here will be key to facilitate more focused patient assessments. The actual use of measures in cancer nursing practice to allow monitoring of PROs must be tested for feasibility and usability.

View all citing articles on Scopus

View full text

‘More of an art than a science’? The development, design and mechanics of the Delphi Technique

Abstract

Section snippets

Author statement

The panel

Origins of the Delphi

Advantages and disadvantages

Conclusion

Declaration of competing interest

J Clin Epidemiol

Technol Forecast Soc Change

Technol Forecast Soc Change

Int J Forecast

Technol Forecast Soc Change

Technol Forecast Soc Change

Technol Forecast Soc Change

Technol Forecast Soc Change

J Clin Epidemiol

Am J Pharmaceut Educ

Res Soc Adm Pharm

Res Soc Adm Pharm

Res Soc Adm Pharm

Eur J Integr Med

Int J Res Market

Technol Forecast Soc Change

Inf Manag

Technol Forecast Soc Change

Technol Forecast Soc Change

Technol Forecast Soc Change

RAND/UCLA Appropriateness Method User's Manual

How to use the nominal group and Delphi techniques

Int J Clin Pharm

STOPP/START criteria for potentially inappropriate prescribing in older people: version 2

Age Ageing

A modified Delphi process to define competencies for assessment leads supporting a doctor of pharmacy program

Am J Pharmaceut Educ

Qualitative Research: reaching the parts other methods cannot reach: an introduction to qualitative methods in health and health services research

BMJ

The Delphi Method - Techniques and Applications

The impact of the internet on community pharmacy practice: a comparison of a delphi panel's forecast with emerging trends

Health Market Q

Research guidelines for the Delphi survey technique

J Adv Nurs

The Delphi Technique: A Methodology to Support the Development of a National Set of Child Well-Being Indicators

Qualitative Research: consensus methods for medical and health services research

BMJ

The development of an accreditation framework for continuing education activities for pharmacists

Pharmacy

Defining pharmacists' roles in disasters: a Delphi study

PloS One

Core outcome set for trials aimed at improving the appropriateness of polypharmacy in older people in primary care

J Am Geriatr Soc

PIPc study: development of indicators of potentially inappropriate prescribing in children (PIPc) in primary care using a modified Delphi technique

BMJ Open