Review
From concepts to comparisons: A resource for diagnosis and measurement in social-ecological systems

https://doi.org/10.1016/j.envsci.2020.02.009Get rights and content

Highlights

  • Inconsistencies in variable selection and measurement limit social-ecological studies.

  • These limitations relate to a suite of methodological challenges.

  • A new variables repository helps users diagnose and measure key factors.

  • Through this resource more comparable research can be done across cases.

Abstract

A central challenge facing the study the environmental governance is the lack of commonunderstanding of important concepts. Critical concepts such as social boundaries, property rights, and resource dependence are selected and measured inconsistently across research projects and field settings, producing results that are difficult to compare. This stymies the accumulation of scientific evidence regarding the most effective ways to address challenging environmental problems. As members of the Social-ecological systems meta-analysis database (SESMAD) project, we have addressed this challenge by developing a repository of variables associated with many of the most important concepts across a range of fields related to environmental governance. In this paper we describe the infrastructure behind the repository, the range of variables it includes, and how it can enable scholars across a range of fields to more systematically select and measure the variables to include in their analyses.

Introduction

There are diverse approaches for studying human-environment interactions, including conservation biology, institutional analysis and political ecology. Research across these fields involves many empirical factors that characterize complex, real-world settings (Agrawal, 2003; Liu et al., 2007). A central challenge facing these fields is the lack of a common understanding regarding the meaning of key concepts and the set of relevant variables (Pullin, 2015). For example, leadership, a key concept across multiple fields, has been viewed by some collective action scholars as occurring when actors make significant contributions to the provision of public goods (Glowacki and von Rueden, 2015). Alternatively, other scholars have equated leadership to certain positions, degree of influence in a community, or certain socio-demographic features like education or wealth (Meinzen-Dick et al., 2002; Villamayor-Tomas et al., 2014; Vedeld, 2000). While some studies point to the special skills of leaders to promote institutional development and adaptation (Meinzen-Dick et al., 2002; Olsson et al., 2004), others point to their privileged position to change rules to their own advantage (Andersson and Ostrom, 2008). Results from empirical studies on the importance of leadership are therefore mixed.

This lack of common understanding leads to inconsistent results in several ways (see Araral, 2014 and Cox et al., 2016a, 2016b for a broader discussion). First, in quantitative observational work, variable choice can be highly idiosyncratic. Standard protocols for describing why some variables and not others are included in empirical models generally do not exist in the literature. When hypotheses are included to motivate the variables that are included, this is usually not accompanied by a discussion of why many others were not included. Combining this with the plethora of variables that are known to be relevant can lead to challenges in the specification of empirical models. From a quantitative perspective, it leads to the likelihood of committing two analytical errors: (1) including variables that we should exclude, and (2) excluding variables that we should include. Including too many variables can rapidly erode statistical power, and lead to overdetermination, multi-collinearity and post hoc theorizing of coincidentally significant results that cannot be reproduced. Including too few variables, meanwhile, can bias estimates of the effects of variables through endogeneity. Finally, when the reasons for variable choice are underdeveloped and thus not standardized, this can leave more room for confirmation biases to affect methods and results.

In addition to variable selection, the measurement of included variables can be idiosyncratic as well. When differences in findings are artifacts of varying measurement protocols, we cannot be sure that comparisons across study sites and projects are meaningful. Without comparable results, we cannot accumulate a core set of established facts, the hallmark of successful scientific research programs and sound policy advice (Pullin, 2012).

Addressing these challenges to produce consistent information across empirical settings poses an immense collective action problem as it requires researchers to coordinate their efforts to consistently define and measure concepts. Ostrom (2007, 2009) social-ecological system (SES) framework was ostensibly designed to address coordination problems in social-ecological research by providing scholars with a common language for empirical inquiry. A key part of Ostrom’s argument was that a “diagnostic approach”, based on the arrangement of variables along multiple tiers, would help scholars decide what mattered in a particular context. This was designed to help deal with the problems outlined above, particularly with respect to the plethora of relevant variables. This approach has been elaborated by others (Young et al., 2018), suggesting the development of a “diagnostic toolkit”, but to date few resources are available that enable the diagnosis of SESs in the way that Ostrom envisioned.

Furthermore, although the SES framework provides a useful entry point for research and analysis, in a recent review of applications of the SES framework, Partelow (2018) highlights the lack of “general methods, guidelines or procedures”, which has resulted in the continued use of inconsistent definitions, indicators and measures. As a result, Ostrom’s framework has not contributed to advances in cumulative understanding as much as many had hoped (Thiel et al., 2015; Schlager and Cox, 2017).

In this paper we present an effort to address this suite of problems in the form of a repository of social-ecological variables, which could form the basis for a diagnostic, social-ecological toolkit. This was collectively developed by the authors to address challenges in selecting, defining and measuring a set of variables associated with well-established concepts relevant for the study of human-environment interactions, as well as providing tools for case comparison (https://sesmad.dartmouth.edu/variables). We developed the variables repository as a part of the social-ecological systems meta-analysis database (SESMAD) project, which built upon Ostrom’s SES framework (Cox, 2014). The SESMAD project was originally developed to systematically code and compare large-scale commons (Ban et al., 2017; Davies et al., 2018; Fleischman et al., 2014). It has, however, also been applied to the study of small periodically harvested closures in Fiji (Jupiter et al., 2017) and supported a synthetic summary of important theories of natural resource governance (Cox et al., 2016a). While the variables we included were intended to support our specific project goals, we hope that it will be built upon and expanded by interested scholars. No similar repository exists to address the challenges of variable identification, definition, measurement for the study of human-environment interactions, although some similar efforts exist for specific fields or sectors: e.g. Salafsky et al. (2008) (http://www.conservationmeasures.org/); Wollenberg et al. (2007) and Chhatre and Agrawal (2009) (ifriresearch.net). Below we present our methods for constructing the repository, describe the variables it contains, how it can be used to implement Ostrom’s diagnostic framing, and discuss its limitations and implications for future work.

Section snippets

Methods

Identifying and defining variables relevant to the study of human-environment interactions required several steps. The first of these was to identify scientific concepts across the relevant literatures. Scientific concepts reflect theoretically important ideas and narratives without explicit reference to measurement, whereas variables are associated with a well-defined range of possible values across a range of observations (Adcock, 2001). For this step, we first identified scientific concepts

Results

To date we have identified and defined 177 variables in the SESMAD database to achieve the goals of the SESMAD project. All variables and their definitions and references are available on the SESMAD website: https://sesmad.dartmouth.edu/variables. Each is defined by values assigned across a set of fields in the SESMAD database variables table (Table 1).

For the purposes of comparative analysis and given the available empirical data, most variables were measured at the ordinal and categorical

Variable choice and diagnosis

The repository of variables (https://sesmad.dartmouth.edu/variables), while not exhaustive, reflects the number and diversity of concepts and variables that have been introduced in the study of human-environment interactions. As we have discussed, this diversity presents a great analytical challenge, to which Ostrom’s diagnostic framework was an attempted solution.

Here we describe how the variables repository represents an attempt at helping users diagnose the important factors to consider in

Conclusions: limitations and further development

Our effort to identify and define variables in order to facilitate comparative research and theory development has several limitations. First, while we define variables, we do not provide instructions for data collection and inference regarding the values of each variable, nor do we develop specific guidelines for selecting variables, beyond what the structure of database provides as just described. Guidance for such variable selection and data collection would be useful, including outlining

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References (40)

  • C.R. Binder et al.

    Comparison of frameworks for analyzing social-ecological systems

    Ecol. Soc.

    (2013)
  • S. Carpenter et al.

    From metaphor to measurement: resilience of what to what?

    Ecosystems

    (2001)
  • A. Chhatre et al.

    Trade-offs and synergies between carbon storage and livelihood benefits from forest commons

    PNAS

    (2009)
  • F. Clement

    Analysing decentralised natural resource governance: proposition for a “politicised” institutional analysis and development framework

    Policy Sci.

    (2010)
  • M. Cox

    Understanding large social-ecological systems: introducing the SESMAD project

    Int. J. Commons

    (2014)
  • M. Cox et al.

    A review of design principles for community-based natural resource management

    Ecol. Soc.

    (2010)
  • T.E. Davies et al.

    Assessing trade-offs in large marine protected areas

    PLoS One

    (2018)
  • G. Epstein et al.

    Missing ecology: integrating ecological perspectives with the social-ecological system framework

    Int. J. Commons

    (2013)
  • G. Epstein et al.

    Governing the invisible commons: ozone regulation and the Montreal Protocol

    Int. J. Commons

    (2014)
  • G. Epstein et al.

    Into the deep blue sea: Commons theory and international governance of Atlantic Bluefin Tuna

    Int. J. Commons

    (2014)
  • Cited by (0)

    View full text