A delphi-style approach for developing an integrated food/non-food system sustainability assessment tool
Introduction
The concept of sustainability is not a concrete one. Whilst attempts have been made to develop a universal framework for its assessment – the FAO's ‘Sustainability Assessment of Food and Agriculture systems’ (SAFA) for example – the appropriate indicators, the benchmarks against which to judge them and even the components important for a sustainable system remain very much determined by the developers' personal values and beliefs. It is perhaps for this reason, alongside the researcher-perspective of many assessments and frameworks, that despite large numbers of options, the voluntary application of sustainability assessments by producers is rare.
One way to overcome the subjectivity caused by researcher decision-making in areas where rigorous scientific analysis or experimentation is not possible is to adopt a participatory or bottom-up approach to decision making (Delbecq et al., 1975 in Saint-Germain et al., 2000; Merfield et al., 2015; Schmitt et al., 2017, Rose et al., 2018). Participatory approaches to research are increasing, particularly in fields such as audits and assessments where the outcomes of the research will have to be adopted directly by stakeholders. Besides the belief that drawing on the knowledge of many increases the accuracy and relevance of the answer (Dalkey, 1969), there is evidence that adoption of a participatory approach increases subsequent perceived usefulness of the research outputs (deLancer Julnes, 2001).
A popular method for participatory research is the Delphi approach (Mukherjee et al., 2015). Whilst the exact implementation varies between projects, the approach can be identified by three integral principles – anonymous response, iteration and controlled feedback – and statistical group response. However, the Delphi approach as typically implemented poses a number of challenges, especially where the project and its stakeholders extend across multiple countries and represent stages of the value chain with potentially very different perspectives. As one example, the authority of the researcher can adversely affect the process, leading to bias in the wording of questions or the selection of experts (Avella, 2016). Researchers may similarly impose their preconceptions on the respondents, particularly where responses are sought to criteria identified in a literature review conducted by the same researcher or team (ibid, 2016). There can also be a tendency to become bogged down in discussions/debates over the method rather than the topic, whilst low response rates and inconsistent concept descriptions can lead to practical challenges with regard to data reliability and comparability (ibid, 2016).
This work describes the design and application of a Delphi-style approach for application in an international research project investigating the role of integrated food/non-food production in the design of more sustainable farming systems (SustainFARM – www.sustainfarm.eu). The method we develop, although applied here to sustainability indicators for agroforestry, provides a new way to integrate literature review, questionnaire-based Delphi and face-to-face Delphi approaches, in a way accessible to projects operating cross-border, cross-country and within a restricted time-frame and budget. The goal of the research presented is therefore to assess the suitability of this modified Delphi approach for achieving consensus on sustainability indicator selection within an international and multi-stakeholder research context.
Within the SustainFARM project, the Delphi approach was adopted to identify sustainability indicators for integrated production systems for use in the modification of an existing, agriculture-focused sustainability assessment tool. In this context, integrated food/non-food production was considered to include the use of waste residues from the agricultural system, as well as the use of incidental (e.g. hedgerows) or cultivated (e.g. tree rows in alley cropping) woody biomass, i.e. agroforestry.
The sustainability assessment tool selected for modification was the Public Goods tool (PG Tool, Gerrard et al., 2012), originally developed through two Defra-funded pilot projects that sought to combine data from scientific literature with expert-input in order to produce a framework for the evaluation of a farm's environmental, economic and social performance.
In the following section, we describe the modified-Delphi approach applied to select new indicators for the adaptation of the PG tool for agroforestry systems in Europe. We then go on to present the results of this exercise, both for the research itself and the performance of the method.
Section snippets
Methodology
Six national partners participated in this Delphi process: University of Copenhagen, Denmark – DK; CNR, Italy – IT; IUNG-PIB, Poland – PL; Phillipps-University Marburg – DE; UASVMCN, Romania – RO and Progressive Farming Trust (trading as The Organic Research Centre), United Kingdom – UK. Each contributed to different stages of the process shown in Fig. 1 below.
The first stage of the Delphi was to identify the pool of ‘experts’ whose knowledge would be drawn upon. Stakeholder categories were
Results
127 stakeholders contributed their expertise to this research: 25 as part of the literature search/consultation phase and 102 during the Delphi process (Table 3). There was a high drop-out rate at all stages and a tendency for biased coverage of stakeholder categories within each country. However, overall, coverage of all stakeholder groups was achieved (Table 3).
Discussion
The conventional Delphi method has a number of limitations. Time and cost often makes it prohibitive from an organisational perspective (Bamberger and Mair, 1976), whilst for more complex or extensive data sets, retaining stakeholder engagement and engaging with more ambiguous issues can be a challenge. The method detailed in this study manages to address many of these issues, successfully reducing a long list of complex indicators to a much shorter list of indicators known to be relevant to
Conclusions
The Delphi process is a key research method where ambiguity and complexity persists, perspectives and interests differ and there is no ‘right’ answer. Yet what reflects ‘good practice’ in a Delphi is much debated and the method is currently associated with many limitations that make it hard to apply in any situation, but particularly in the circumstances of high complexity and diverse interests that it is best suited to.
This study takes one such area of research – sustainability indicators for
Acknowledgements
This work has been completed as part of the FACCE SURPLUS project ‘SustainFARM’ (http://www.sustainfarm.eu/). SustainFARM is a three year project, funded in the UK by Defra as part of the European FACCE SURPLUS ERA-NET co-fund programme formed in collaboration between the European Commission and a partnership of 15 countries. FACCE SURPLUS has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 652615. Thanks also go to the
Declaration of Competing Interest
None.
References (24)
- et al.
Comparing face-to-face meetings, nominal groups, Delphi and prediction markets on an estimation task
Int. J. Forecast.
(2011) - et al.
The effect of controlled opinion feedback on Delphi features: mixed messages from a real-world Delphi experiment
Technol. Forecast. Soc. Change
(2016) - et al.
Comparing the sustainability of local and global food products in Europe
J. Clean. Prod.
(2017) Delphi panels: research design, procedures, advantages, and challenges
International Journal of Doctoral Studies
(2016)- et al.
Die Delphi-Methode in der Praxis: Ergebnisse einer exploratorischen Untersuchung zu Einsatzbereichen und Anwendererfahrungen
Manag. Int. Rev.
(1976) - et al.
Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review
PLoS One
(2011) The Delphi Method: An Experimental Study of Group Opinion
(1969)Does participation increase perceptions of usefulness? An evaluation of a participatory approach to the development of performance measures
Public Perform. Manag. Rev.
(2001)- et al.
Group Techniques for Program Planning: a Guide to Nominal Group and Delphi Processes
(1975) Sustainability Assessment of Food and Agriculture Systems (SAFA) Guidelines
(2013)
Public goods and farming
Data saturation [online; accessed march 2020]
Cited by (17)
Evaluating the environmental impact of agri-food waste reduction: A bibliographic analysis and conceptual framework
2024, Environmental Impact Assessment ReviewIntegrating the social perspective into the sustainability assessment of agri-food systems: A review of indicators
2023, Sustainable Production and ConsumptionDefining common criteria for harmonizing life cycle assessments of livestock systems
2023, Cleaner Production LettersAssessing the multidimensional elements of sustainability in European agroforestry systems
2022, Agricultural SystemsCitation Excerpt :The PG tool was selected as the overarching method due to its coverage of a broad range of sustainability criteria (FAO, 2013; Gerrard et al., 2012; Paraskevopoulou et al., 2020) and due to its adaptability. Within this study the ‘default’ version of the PG tool (Gerrard et al., 2012) was adapted for AF systems through the incorporation of agroforestry-specific sustainability indicators (Appendix A: Supplementary Material, S1, Mullender et al., 2020). Candidate indicators for the adaptation were selected through a structured literature review of previous studies that have assessed the sustainability of AF systems in Europe.
- 1
Permanent address: Anthesis Group, 9 Newtec Place, Oxford, OX4 1RE, United Kingdom.
- 2
Permanent address: MV Agroecological Research Centre, Moinhos de Vento, 7750-217 Espirito Santo, Portugal.
- 3
Permanent address: School of Agriculture, Food and Environment, Royal Agricultural University, Cirencester, GL7 6JS, United Kingdom.
- 4
Permanent address: PricewaterhouseCoopers GmbH Wirtschaftsprüfungsgesellschaft, Moskauer Straße 19, 40227, Düsseldorf.