A delphi-style approach for developing an integrated food/non-food system sustainability assessment tool

https://doi.org/10.1016/j.eiar.2020.106415Get rights and content

Highlights

  • A well-designed Delphi can operate effectively across regions and expertise.

  • Environment indicators are less controversial than other indicators of sustainability.

  • Face-to-face discussion and reasoning helps aid consensus and settle viewpoints.

Abstract

Sustainability assessment is a complex field and its uptake amongst agricultural producers limited. Furthermore, the scope of current sustainability assessment tools does not extend to systems in which food production is integrated with production of non-food biomass (e.g. agroforestry). Participatory approaches to tool development offer a means to overcome the subjectivity of researcher-led tool design and thus the potential to increase relevance and engagement. In this work we develop a Delphi-style methodology as a means to produce a sustainability assessment tool suitable to assess and feedback on an integrated food/non-food system. Using a widely accepted agricultural sustainability framework and an existing farm sustainability assessment tool as a base, stakeholders were engaged with across six countries and multiple stakeholder groups to identify key indicators to be added to the tool. The methodology developed is described in detail, framed in the setting of this tool development process but providing a novel framework applicable to any situation where indicators must be developed for a complex issue of interest across multiple perspectives and stakeholder groups. Feedback and learning from the experience is provided. It was found that, contrary to some opinion, the inclusion of a face-to-face discussion round as part of the Delphi procedure provides a valuable means for information exchange and a move towards consensus amongst stakeholders. By using a ‘snowball’ approach to the in person discussions, it appears too that the loss of the voices of more socially retiring individuals can be avoided. Final levels of agreement vary substantially across the different areas of sustainability, with indicators in some areas (e.g. environmental integrity) proving much less controversial than others (e.g. social wellbeing). Despite this, the methodology effectively reaches a level of consensus amongst diverse stakeholders sufficient to guide the selection of sustainability indicators with a good level of confidence.

Introduction

The concept of sustainability is not a concrete one. Whilst attempts have been made to develop a universal framework for its assessment – the FAO's ‘Sustainability Assessment of Food and Agriculture systems’ (SAFA) for example – the appropriate indicators, the benchmarks against which to judge them and even the components important for a sustainable system remain very much determined by the developers' personal values and beliefs. It is perhaps for this reason, alongside the researcher-perspective of many assessments and frameworks, that despite large numbers of options, the voluntary application of sustainability assessments by producers is rare.

One way to overcome the subjectivity caused by researcher decision-making in areas where rigorous scientific analysis or experimentation is not possible is to adopt a participatory or bottom-up approach to decision making (Delbecq et al., 1975 in Saint-Germain et al., 2000; Merfield et al., 2015; Schmitt et al., 2017, Rose et al., 2018). Participatory approaches to research are increasing, particularly in fields such as audits and assessments where the outcomes of the research will have to be adopted directly by stakeholders. Besides the belief that drawing on the knowledge of many increases the accuracy and relevance of the answer (Dalkey, 1969), there is evidence that adoption of a participatory approach increases subsequent perceived usefulness of the research outputs (deLancer Julnes, 2001).

A popular method for participatory research is the Delphi approach (Mukherjee et al., 2015). Whilst the exact implementation varies between projects, the approach can be identified by three integral principles – anonymous response, iteration and controlled feedback – and statistical group response. However, the Delphi approach as typically implemented poses a number of challenges, especially where the project and its stakeholders extend across multiple countries and represent stages of the value chain with potentially very different perspectives. As one example, the authority of the researcher can adversely affect the process, leading to bias in the wording of questions or the selection of experts (Avella, 2016). Researchers may similarly impose their preconceptions on the respondents, particularly where responses are sought to criteria identified in a literature review conducted by the same researcher or team (ibid, 2016). There can also be a tendency to become bogged down in discussions/debates over the method rather than the topic, whilst low response rates and inconsistent concept descriptions can lead to practical challenges with regard to data reliability and comparability (ibid, 2016).

This work describes the design and application of a Delphi-style approach for application in an international research project investigating the role of integrated food/non-food production in the design of more sustainable farming systems (SustainFARM – www.sustainfarm.eu). The method we develop, although applied here to sustainability indicators for agroforestry, provides a new way to integrate literature review, questionnaire-based Delphi and face-to-face Delphi approaches, in a way accessible to projects operating cross-border, cross-country and within a restricted time-frame and budget. The goal of the research presented is therefore to assess the suitability of this modified Delphi approach for achieving consensus on sustainability indicator selection within an international and multi-stakeholder research context.

Within the SustainFARM project, the Delphi approach was adopted to identify sustainability indicators for integrated production systems for use in the modification of an existing, agriculture-focused sustainability assessment tool. In this context, integrated food/non-food production was considered to include the use of waste residues from the agricultural system, as well as the use of incidental (e.g. hedgerows) or cultivated (e.g. tree rows in alley cropping) woody biomass, i.e. agroforestry.

The sustainability assessment tool selected for modification was the Public Goods tool (PG Tool, Gerrard et al., 2012), originally developed through two Defra-funded pilot projects that sought to combine data from scientific literature with expert-input in order to produce a framework for the evaluation of a farm's environmental, economic and social performance.

In the following section, we describe the modified-Delphi approach applied to select new indicators for the adaptation of the PG tool for agroforestry systems in Europe. We then go on to present the results of this exercise, both for the research itself and the performance of the method.

Section snippets

Methodology

Six national partners participated in this Delphi process: University of Copenhagen, Denmark – DK; CNR, Italy – IT; IUNG-PIB, Poland – PL; Phillipps-University Marburg – DE; UASVMCN, Romania – RO and Progressive Farming Trust (trading as The Organic Research Centre), United Kingdom – UK. Each contributed to different stages of the process shown in Fig. 1 below.

The first stage of the Delphi was to identify the pool of ‘experts’ whose knowledge would be drawn upon. Stakeholder categories were

Results

127 stakeholders contributed their expertise to this research: 25 as part of the literature search/consultation phase and 102 during the Delphi process (Table 3). There was a high drop-out rate at all stages and a tendency for biased coverage of stakeholder categories within each country. However, overall, coverage of all stakeholder groups was achieved (Table 3).

Discussion

The conventional Delphi method has a number of limitations. Time and cost often makes it prohibitive from an organisational perspective (Bamberger and Mair, 1976), whilst for more complex or extensive data sets, retaining stakeholder engagement and engaging with more ambiguous issues can be a challenge. The method detailed in this study manages to address many of these issues, successfully reducing a long list of complex indicators to a much shorter list of indicators known to be relevant to

Conclusions

The Delphi process is a key research method where ambiguity and complexity persists, perspectives and interests differ and there is no ‘right’ answer. Yet what reflects ‘good practice’ in a Delphi is much debated and the method is currently associated with many limitations that make it hard to apply in any situation, but particularly in the circumstances of high complexity and diverse interests that it is best suited to.

This study takes one such area of research – sustainability indicators for

Acknowledgements

This work has been completed as part of the FACCE SURPLUS project ‘SustainFARM’ (http://www.sustainfarm.eu/). SustainFARM is a three year project, funded in the UK by Defra as part of the European FACCE SURPLUS ERA-NET co-fund programme formed in collaboration between the European Commission and a partnership of 15 countries. FACCE SURPLUS has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 652615. Thanks also go to the

Declaration of Competing Interest

None.

References (24)

  • C.L. Gerrard et al.

    Public goods and farming

  • Data saturation [online; accessed march 2020]

  • Cited by (17)

    • Assessing the multidimensional elements of sustainability in European agroforestry systems

      2022, Agricultural Systems
      Citation Excerpt :

      The PG tool was selected as the overarching method due to its coverage of a broad range of sustainability criteria (FAO, 2013; Gerrard et al., 2012; Paraskevopoulou et al., 2020) and due to its adaptability. Within this study the ‘default’ version of the PG tool (Gerrard et al., 2012) was adapted for AF systems through the incorporation of agroforestry-specific sustainability indicators (Appendix A: Supplementary Material, S1, Mullender et al., 2020). Candidate indicators for the adaptation were selected through a structured literature review of previous studies that have assessed the sustainability of AF systems in Europe.

    View all citing articles on Scopus
    1

    Permanent address: Anthesis Group, 9 Newtec Place, Oxford, OX4 1RE, United Kingdom.

    2

    Permanent address: MV Agroecological Research Centre, Moinhos de Vento, 7750-217 Espirito Santo, Portugal.

    3

    Permanent address: School of Agriculture, Food and Environment, Royal Agricultural University, Cirencester, GL7 6JS, United Kingdom.

    4

    Permanent address: PricewaterhouseCoopers GmbH Wirtschaftsprüfungsgesellschaft, Moskauer Straße 19, 40227, Düsseldorf.

    View full text