Introduction

Large infrastructure projects are often contested for being over budget or over time because, or being a product of politicians’ ‘monument complex’ (Flyvbjerg et al., 2003; Hall, 1980; Leijten, 2017; Wegrich et al., 2016). In response, international organizations and government think tanks have suggested that decision making should be based on the results of ex ante analyses, such as cost–benefit analyses and environmental effect studies (Ministerie van Infrastructuur en Milieu, 2016; OECD, 2015a, 2015b; World Bank, 2014a, 2014b; World Economic Forum, 2012). The expectation is that ex ante analyses can improve decision making by providing hard evidence on costs, benefits, and effects.

Policy research on knowledge use, however, tells a different story. For example, knowledge ends up not being used or not used for their intended purpose (Feldman, 1989; Weiss, 1979). If it is used, it is because it fit the preconceived positions of dominant actors rather than because of their evidential value (Stevens, 2007, 2011). Moreover, policy research has shown how relying on scientific knowledge tends to depoliticize policy choices that are essentially political. Depoliticization occurs, for instance, when studies demarcate which project dimensions are up for discussion and which dimensions are out of bounds (Flinders & Wood, 2015; Jasanoff, 1990). From this perspective, studies such as the ex ante analysis appear to be at best a ‘boundary object’ (Gieryn, 1995) or a ‘useful myth’ (Boswell, 2017) to overcome policy controversy. Moreover, they are thought to often make policy processes messier instead of providing the promised clarity; it can introduce complex technical procedures that ultimately distract from the political core of the policy process (Parsons, 2002). In some cases, ex ante analysis may even spark the very policy conflicts it is trying to prevent (Wolf & Van Dooren, 2017), or result in a situation in which actors keep producing analyses as counterarguments to the analyses by other actors, resulting in a ‘dialogue of the deaf’ where actors no longer listen to each other’s actual arguments (van Nispen, 2003).

In other words, policy advice and policy research paint widely varying pictures of analysis use. On top of that, policy theories of analysis use differ to a great extent among each other. The aim of this paper is to explain how knowledge us can be depicted in many seemingly contradictory ways. The wide variety of conceptions of knowledge use could mean one of two things: first, it could mean that some of these conceptions are incorrect, and second, it could mean that pieces of knowledge have certain properties that allow it to facilitate many different uses simultaneously. This paper explores this question by studying ex ante analysis as a popular form of knowledge in policy processes.

In the first section of this paper, we show the wide range of theories of analysis use by drawing examples from three key perspectives in the academic debate on the uses of knowledge in policy processes. Then, we describe how we employed ethnographic methods to explain how ex ante analysis are able to reflect all these different theories of use. We immersed ourselves in the preparatory processes of three large infrastructure projects: two in Flanders and one in the Netherlands, during which we paid particular attention to the use of ex ante analysis—a form of analyses which plays a central role in these processes. The ‘Results’ section first describes how the practice of ex ante analysis use does indeed reflect many of the theoretical perspectives mentioned in our theoretical framework. This section goes on to explain the characteristics of ex ante analysis that allow it to be so ‘chameleonic.’ We found that ex ante analysis is able to be chameleonic because of three main characteristics: (1) it employs methods that make an analysis’ outcome relative to its inputs, while its outcomes seem very robust and undisputable, (2) there is no definitive authority deciding which interpretation of an ex ante analysis is correct, (3) trust in ex ante analysis is based on a varied set of criteria. These three characteristics allow ex ante analysis to facilitate a wide variety of discussions without there being a definitive arbiter deciding which interpretation of the analysis is correct. Because trust in ex ante analysis is based on a number of different criteria, trust in analysis is generally high, and people are unlikely to set its results aside easily. As such, ex ante analysis can be used in many different ways, while continuing to appear credible, as is reflected in the variance among theories on knowledge use.

Knowledge in policy processes

Knowledge use has been studied extensively in public administration, policy studies, and organization studies. This has led to a plurality of perspectives, which oftentimes offer conflicting and sometimes even diametrically opposed views of knowledge use. In this section, we group theories of knowledge use in three prominent perspectives. Based on the work of Weiss (1979), we distinguish between linear perspectives, nonlinear perspectives, and critical perspectives.

Linear perspectives

Linear perspectives perceive the relationship between study outcomes and the policy process as direct. In the linear perspective, the results of a study can be directly implemented in the policy process. Weiss (1979) distinguishes between two linear approaches: the knowledge-driven approach and the problem-solving approach. The knowledge-driven approach assumes that the mere presence of knowledge will lead to its use. In the problem-solving approach, knowledge is produced in response to a specific problem. The problem drives the research question, and the research shows how the problem can be solved. The problem-solving approach to the use of evidence in infrastructure decision-making processes is prevalent among advisory reports (Andres et al., 2015; Andres et al., 2007; Brown et al., 2006; European Conference of Ministers of Transport, 2005; OECD, 2012, 2015a, 2015b; World Bank, 2014a, 2014b; World Economic Forum, 2012), government guidelines (‘Decreet betreffende complexe projecten,’ 08-27-2014; Infrastructure and Projects Authority, 2017; Ministerie van Infrastructuur en Milieu, 2016), and management handbooks (Klaassen & Hakvoort, 2015; Priemus & van Wee, 2013; Priemus et al., 2008; Sowden et al., 2011; Taylor, 1947; Wegrich et al., 2016). Handbooks typically discuss knowledge as a means to an end, the ‘end’ being an optimal decision. By directly showing the solution that is allegedly best, ex ante analysis promises to take over politicians’ responsibility for choosing an alternative.

Nonlinear perspectives

Many policy makers do not appear to believe that the linear perspective is an adequate representation of the practice of policy making (Dorren, 2018; Mouter, 2014, 2016). In a self-criticism, Weiss (1979) presents a second group of perspectives which describe the relationship between facts and policy processes as anything but linear. Weiss (1979) distinguishes between three nonlinear perspective models. First, the interactive model argues that knowledge is used in a complex arena of interactions between many different actors among which knowledge-producing scientists are only one of many. Research informs policy, but research findings seldom produce conclusive evidence that points toward the right solution to a problem. The second model Weiss discusses is the political model, which assumes that knowledge is only used when it aligns with a political purpose. In this model, a decision maker will only refer to the results of an ex ante analysis if it supports their preferred decision. The political model is also employed by (Huff, 1991) and Barker and Guy Peters (1993a), who analyze how facts are framed and employed in a power struggle between different actors with competing goals in such a way that each actor achieves their goals. Barker and Guy Peters (1993b), for instance, have edited a set of chapters that describe how scientific expertise is used to legitimize policy. In this case, what fits the policy goal appears to be more important than the quality of the science and the consensus among scientists (Topf, 1993). Finally, the third model is the tactical model, which can be applied to situations where research is being used for reasons that have ‘little relation to the substance of the research’ (Weiss, 1979, p. 429). In this model, research can be used to do things such as delay action or deflect criticism by using a research outcome as the reasoning behind a policy decision. Boswell (2009) presents what appears to be a variation on Weiss’ tactical and political models, where policy makers and politicians refer to research to substantiate policy preferences or to steer a discussion in a particular way. However, Boswell also finds that policy makers actually were reluctant rely too much on scientific research in their decision making, because of the epistemic uncertainties that come with a scientific study. As such, the role of knowledge simultaneously is substantial and marginal.

Weiss’ last and arguably most referred to model is called the enlightenment model. This model holds that research does not directly influence policies; rather, it ‘diffuses circuitously through various channels such as professional journals, the mass media, conversations with colleagues and over time the variables it deals with and the generalizations it offers provide decision makers with ways of making sense out of a complex world’ (Weiss, 1979, p. 430). Instead of influencing decisions directly, research indirectly affects the frame of reference that policy makers use to make decisions. While it is often regarded as most accurately representing the practice of policy making (Marra, 2000), the enlightenment model lacks true explanatory capacity. It merely states that research reaches policy makers a variety of ways, after which some of it ends up being used and some of it does not (Stevens, 2007). The common observation that this model is the most representative of policy making in practice demonstrates the challenge of moving beyond superficial description without also delving deep into the specifics of the case studied. The representativeness of the enlightenment model shows that the ways knowledge influences policy processes is largely case by case. It is likely that, for this reason, other nonlinear perspectives on knowledge use (such as Cairney (2017, 2018); Head (2008); Marmot (2004); Mulgan (2005); Strassheim and Kettunen (2014); Young et al. (2002)) do not replace old typologies, but rather they add to them.

Critical perspectives

The critical perspective is another way to understand knowledge use. In contrast to the previous perspectives, the critical perspective does not primarily concern itself with describing the complexities of policy practice. Rather, it focuses on the underlying power structures and discourse that shapes knowledge use (Triantafillou, 2017, pp. 9–25). It could be argued that the critical project is a project of re-politicization, laying bare power structures and putting them up for debate.

Stevens (2007), for example, uses the concept of ‘survival of the fittest’ to explain which pieces of research influence decisions and which do not (see also: Monaghan, 2009). Policy makers select evidence to craft ‘policy stories’ that fit the dominant narrative within government Stevens (2011). The critical perspective for using research could transform a question about ethics into a financial dilemma that fits a new public management context. Similarly, using this perspective, a debate about whether a certain punishment is fair could be reframed into a debate about the societal costs of that punishment. Stevens also notes that these evidence-based narratives serve the essential function of ‘reduc[ing] the role of uncertainty as a barrier to action’ (Stevens, 2011, p. 234). Policy makers need to make decisions, but they are faced with uncertainty and lack of predictability. Evidence builds confidence. The idea of linear perspectives of knowledge use is therefore what Boswell (2017) calls a ‘useful myth.’ Even though research does not present policy makers with objective truths, it is still useful because it increases confidence and brings people together because it is based on principles most process participants will support (Boswell, 2017).

Other critical perspectives describe the role of knowledge in policy processes as a continuous battle to determine the boundary between science and politics. Whether something is labeled as science or not determines who is able to discuss it (Jasanoff, 1990, p. 236). In order for a critique of something scientific to be taken seriously, it generally has to come from a member of the scientific community. However, the boundary between what is and is not considered science is not set in stone (Gieryn, 1995; Hoppe, 2005). By engaging in what Jasanoff calls ‘boundary work,’ political actors and scientist directly or indirectly negotiate the boundary between policy and science. Subjects of this negotiation process include methods, standards for evidence, and the validity of the interpretation of evidence. Making a similar argument, Grundmann (2009) describes how science plays the role of a referee in policy processes, removing impure—that is, unscientific—elements from a policy process. Parsons (2002) claims that the evidence-based policy movement has made policy processes messier rather than more focused. Policy studies are not isolated from policy processes, but instead are part of it. Demanding that policy processes have to adhere to evidence-based standards frustrates the deliberative processes by neglecting values involved in decision-making and decreasing flexibility. In a sense, these critical perspectives are complementary to linear and nonlinear perspectives. It is possible for a policy maker to literally follow a study outcome, while simultaneously engaging in what can be understood as story-crafting or enforcing a boundary between science and nonscientific knowledge.

It may seem unlikely that ex ante analyses could accommodate multiple, sometimes seemingly opposing perspectives on knowledge use. Those with a linear perspective claim that knowledge gleaned from research can be applied directly to policy processes, but those with a nonlinear or critical perspective problematize this notion. Some may wonder how it is possible that people simultaneously participate in the continuation of a ‘useful myth’ and also use research and analyses to fill knowledge gaps. Based on the fact that all three perspectives appear equally popular, we argue that instead of some perspectives being wrong and others being right, ex ante analysis has certain properties which let it reflect theories from all three perspectives. Therefore, when following a policy process over time and focusing on the messiness of policy practice, one will likely see uses which resemble different theories occur.

Methods and case selection

Because our goal was to gather insights on the development of interactions with ex ante analysis over time without losing sight of the richness of day-to-day policy practice, we decided to use an ethnographic study design. Instead of focusing on a particular person or group that deals with ex ante analysis often, we decided to use particular sets of analyses as our research subject and follow them throughout a policy process for a period of approximately one year. This allowed us to see how different perspectives on the use of analysis interacted and how interpretations of analyses changed over time. The ethnographic design enabled us to witness analysis used in many different situations by a broad range of actors.

Furthermore, the ethnographic design allowed us to move beyond participant reconstruction. What policy makers present in interviews is often a summarized or idealized and coherent version of their actual practice (Czarniawska, 1997; Dorren, 2018; Portelli, 1991). Being present for the analysis enabled us to capture the complexity of interactions between an analysis and a policy process. We maximized context sensitivity (Ybema et al., 2009, pp. 6–7), which is so important according to the enlightenment model by Weiss (1979).

Case selection

We selected three infrastructure policy processes on three projects as cases. In infrastructure policy making, the use of ex ante analysis is relatively highly formalized and extensive (‘Decreet betreffende complexe projecten,’ 2014; Ministerie van Infrastructuur en Milieu, 2016). For instance, in the Netherlands in 2008, the average infrastructure project undertaken by the national government took five years of studying before work on the final design started (Elverding et al., 2008).

We selected cases in the Netherlands and Flanders.Footnote 1 Although the Netherlands neighbors the Flanders region of Belgium, both areas differ significantly in terms of administrative culture. The Netherlands is ‘commonly regarded as one of the strongholds of policy analysis’ (van Nispen & Scholten, 2015) with a longstanding tradition of integrating analyses into policy processes; it also has a range of independent agencies that evaluate policies. Flanders has a culture in which policy decisions are ‘primarily the results of political bargaining,’ which hinders their ability to develop a strongly institutionalized culture of policy analysis (Fobé et al., 2017, p. 51). Including cases from these two diverging contexts reduced the likeliness that the mechanisms we observed, were specific to a particular policy context. Table 1 gives an overview of the cases in this study at the start of the observation process.

Table 1 Cases and characteristics
Full size table

When studying these policy process cases, we paid particular attention to the ways in which process participants discussed ex ante analysis. An ex ante analysis aims to predict. It departs from a reference situation, mostly consisting of the situation ‘which exists at the time the option will be realized, possibly combined with the situation which would arise if the option is not realized’ (Klaassen & Hakvoort, 2015) and then predicts the expected development of certain indicators as opposed to the reference situation. Ex ante analyses are generally used to compare policy options to inform policy choices. They can come in many shapes and sizes: in this study, we encountered everything from computer simulations to qualitative interviews with experts (reported as ‘expert judgments’), all identified as ex ante analyses. One thing that all ex ante analyses had in common was that they involved quantification in some capacity. For example, a cost–benefit analysis predicted the economic costs and benefits of a project in euros, and an environmental impact assessment predicted the environmental impact of a project on a scale from − 3 to + 3.

We conducted observations of these cases between November 2017 and July 2019. At the beginning of our observation period, all projects were in an explorative stage. In the explorative stage, the project aims are broadly laid out. The goal of this phase is to explore and compare several ways of reaching the project’s aims. We had access to two types of meetings: publicly accessible meetings and internal meetings. All cases had publicly accessible meetings. These consisted of, for instance, information markets about the project or more focused participation sessions in which a wide range of actors were asked to give their input. For two of the three cases, we also had access to internal meetings. These consisted of, for example, regular meetings between members of the project staff, one-time presentations by experts, or meetings with the political steering committee of the project. In total, we conducted approximately 200 h of observations in 74 meetings.

A wide variety of actors attended these meetings, from interested citizens to members of action groups and from representatives of local business associations to NGO’s. Table 2 contains an overview of these actors and their interests.

Table 2 Actor types encountered during observations and their interest in the project in question
Full size table

Data collection

Observations were recorded in field notes. Because the meeting settings were often static, the primary focus of these field notes was on what people said and how they interacted. We paid attention to tone of voice and nonverbal communication. In addition, our field notes contained details on setting. They described the look and feel of the rooms in which meetings were held, their location, and any other circumstances of note.

Because of the technical nature of infrastructure projects and the decision-making processes that surround them, we engaged in ‘pre-member checking’ through small conversations with participants during meeting breaks. This enabled us to test our understanding of what was going on in the meetings.

In addition to the meetings, we also had access to a portion of the e-mail communications for each project case. We were included in the email list that was used to circulate the documents that were up for discussion in the meetings. Even though these communications were not explicitly included in our analysis, they did enrich our understanding of the content of the meetings.

Coding

In order to be able to process the data we gathered and arrive at an overview of the uses of ex ante analysis in infrastructure decision-making processes, we devised an inductive coding strategy based on Miles and Huberman’s (1994) strategy for a cross-case comparison. We used NVivo 12 as our coding software. Our strategy consisted of two main analytical steps. First, we conducted a within-case analysis of each case, coding each case separately. This first round of coding began by assigning chunks of data to descriptive codes containing little interpretation (Miles & Huberman, 1994, p. 57). These codes allowed us to distinguish between cases, people, and topics of conversation or observation. This way, we were able to check whether certain observations only held for certain groups of people, or whether there was a difference in the way in which people discussed different aspects of an analysis. Second, we analyzed the content of these descriptive codes in a second round of coding, creating analytical codes. Analytical codes have a more interpretive character (Miles & Huberman, 1994, p. 57). In this particular study, the analytical codes mainly were meant to convey actors’ use and interpretation of ex ante analysis. To be able to explain how ex ante analyses are able to reflect many different models of use, we devised a data-driven coding scheme based on the way in which people discussed specific parts of ex ante analysis. This way of coding allowed us to observe patterns to analyses use beyond the applicability of theoretical perspectives on knowledge use. Our coding resulted in codes such as ‘downplaying,’ for instances in which actors tried to convince other actors to not take analysis outcomes to seriously, or ‘positive,’ capturing a set of reactions in which actors conveyed positive sentiments about an analysis. Other examples of codes were ‘absolutizing’ and ‘relativizing,’ for reactions that, respectively, took study outcomes as a given or as relative to a study’s input.

From codes to perspectives on knowledge use

We used our codes to qualitatively assess whether the theoretical perspectives on knowledge use occurred in our data. For example, we associated the code ‘absolutizing’ with linear perspectives on knowledge use. When assessing whether the observations coded under ‘absolutizing’ contained examples of people employing a linear perspective on knowledge use, we did indeed find statements such as ‘this cost–benefit analysis shows us what the best trajectory for this bike path is’ or ‘we will not continue pursuing [this policy option], as our studies have shown it is unlikely to have any effect.’ We looked at contextual information to assess whether these statements actually most closely resembled a linear perspective, or—for example—a situation more closely resembling a political perspective in which a linear-seeming statement was made.

By comparing our coding to our theoretical framework in this way, we were able to verify whether it is indeed the case that a broad range of theories applies to analysis use, rather than some of these theories being wrong and others being right. By studying patterns occurring across cases in this light, we ultimately distilled a set of properties which allowed ex ante analysis to be used in such varying ways.

Results

In this section, we exemplify how the different perspectives on knowledge use explored above occured in our data. Then, we explain what specific characteristics of ex ante analysis allows it to reflect such a wide variety of perspectives without it resulting in so much ambiguity that leads to the ex ante analysis losing credibility or being abandoned all together. The first characteristic of ex ante analysis we discuss is its built-in ambiguity, which allows the analysis to reflect a multitude of different ideas for its use. The second characteristic we discuss is lack of full ownership of the analysis by any one person, which also means that no one is completely responsible for it. The third characteristic is trustworthiness: people do not seem to lose trust in ex ante analysis results because of, among other things, its complexity and ability to seemingly separate facts from values.

Facilitating many different perspectives

We observed all perspectives of knowledge use during our fieldwork, albeit some were more clearly present than others. A substantial amount of the discussion about analysis we observed expressed a linear perspective on knowledge use. This perspective suggests that a study outcome could directly and exclusively inform parts of a policy outcome. The frequency with which linear perspectives were expressed can be at least partially attributed to legal rules and procedures. For instance, Flanders has air quality norms that limit the negative impact of a project on air quality. When the expected amount of pollution exceeds a certain level, the government is legally required to take measures to decrease pollution. For many people involved in policy processes, such a linear application of knowledge is ideal. In Flanders, an often-heard claim was that people wanted the policy process to be ‘objective’ instead of ‘political.’ This implies the opinion that analyses can point to an objectively right solution, and can, in a way, replace politicians’ input when it comes to making decisions.

In addition to the linear perspective, we also encountered all nonlinear perspectives on knowledge use during our observations. For instance, in one of the project cases, a new policy option was added to an already existing set which had been heavily debated in a deliberation trajectory of almost two years. The new option was announced by a minister and came as a surprise to most participants in the process. The civil servant presiding the participation process was not able to pinpoint where the idea came from, and they were only able to say that it emerged from ‘new insight,’ which would be an example of Weiss’ (1979) enlightenment model.

We also encountered critical perspectives in our observations. Even though these perspectives are of a more analytical nature—meaning one would not expect them to be directly deployed by process participants—we did occasionally observe them in action. For example, the idea of the ex ante analysis as a story-crafting device was mentioned explicitly by members of one of the project management teams. When discussing the design of an ex ante analysis, these people wanted the analysis being able to tell ‘the story of the project.’ In a presentation for project managers in the same case, an analyst proclaimed they wanted their measurements to ‘enable politicians to tell a story’ rather than just having them speak about the benefits of the project in broad terms. Other examples of how different theories of knowledge use appeared in our data, can be found in Table 3.

Table 3 Perspectives on analysis use and examples from data
Full size table

It is remarkable that none of the perspectives can be put aside based on our observations, despite some of them appearing to be mutually exclusive. This means that apparently, ex ante analysis possesses certain qualities that enable an ex ante analysis to reflect theories from all these perspectives, rather than it being the case that some perspectives are more right than others. In the next section, we describe how certain features of ex ante analysis allow it to reflect different theories of use and enable actors in policy processes to switch between different uses.

Like a chameleon: how ex ante analysis reflects many different theories of use simultaneously

We found that ex ante analysis is able to reflect many different theories of use because of three key characteristics. First, ex ante analysis can be relativized, yet deliver absolute-seeming results. Second, the responsibility for ex ante analysis is split up over different actors, up to the extent that outcomes are ascribed to the analysis itself, rather than the people involved in its construction. Lastly, ex ante analysis draws from many different sources of trust.

Facilitating debate: soft methods, hard facts

The first characteristic that allows an analysis to reflect many different perspectives stems from the way the analysis process is structured and the format in which outcomes are presented. As a feature of their design, analyses’ models can be relativized, yet their outcomes have a very absolute character. This causes people to simultaneously relativize and absolutize analyses’ outcomes.

This observation came about as follows. In our analysis, we coded for the level of certainty with which people make statements about analyses or based on analyses. Statements coded as ‘relative certainty’ implicitly or explicitly pointed toward the margining of uncertainty that comes with the outcomes of an analysis. For example, responses pointing at the limitations of a model were coded as expressing relative certainty. Statement coded as conveying ‘absolute certainty’ presented analysis outcomes as if this margin of uncertainty was not a factor. These statements mostly concluded that ‘the analysis shows that x causes y,’ without addressing the uncertainties contained in this outcome. Table 4 shows that when analyzing statements about the analysis made by different groups of people, all of these groups made statements in both categories. In fact, when zooming in on specific individuals within groups, we see that individuals regularly absolutized and then relativized analysis outcomes in the same discussion.

Table 4 Distribution of the level of certainty over statements made by different groups of participants
Full size table

This pattern of behavior is a result of the way in which people react to a specific set of properties of ex ante analysis. An ex ante analysis can be relativized because it is a simplified model of reality. Decisions about what to include in the analysis and what to exclude from it must be made. In all policy processes we observed, there was room for the process participants to provide input about what should be included in and excluded from the analysis, even though some analyses were based on standardized perspectives and indicators. If not already present, inviting actors to provide input for the analysis appeared to spark awareness of the simplifying nature of ex ante analysis.

Where the simplifying nature of ex ante analysis allows for relativizing comments, the outcomes of the analyses have an exact-seeming nature. Outcomes presented appear as solid facts with that are difficult to argue with. They appear precise and not at all the product of a limited set of methods, as is illustrated by the following observation:

The analyst tells the group that they have done computer simulations to determine the optimal shape of the dock. In this simulation a digital ship enters a digital dock under certain conditions to judge the operational quality of each policy option. The analyst tells us that in this case, the tugboats pulled over 80 tons and used x amount of power. They go on, telling us how many times the tugboats had to use which engines and how strong the winds were during these manoeuvres. On the projector screen, a ship-shaped figure moves down a river on what looks like a Google Maps satellite image. It makes a sweeping move across the [river] and parks itself on a couple of fields, in the midst of which resides what appears to be a small house.Footnote 2

The analyst quoted above talks as if they observed real-world tugboats tugging a ship over a river in certain wind conditions. In another presentation on the same study, effects on high tide levels were presented in terms of centimeters. A large results table presenting the outcomes of a cost–benefit analysis in another case displayed the cost differences between project options in eurocents, even though much of its input came from interviews with transport companies. A positive effect on inland waterway shipping in the same project is represented by an absolute number (+ 1), despite it being an estimation based on conversations with government employees and transport companies. The language used to present the outcomes of an analysis often has a confidence and certain tone. They include phrases such as ‘we see that these effects are negligible’ (as if the analyst has seen true evidence of this) or ‘the effects are similar for all alternatives’ (as if the alternatives have been built in reality and the analyst was able to observe them) rather than phrases such as ‘the model shows...’ or ‘’we expect that...’.

These shifting levels of certainty allow process actors to shift between uses of an analysis which would fit under different academic theories on knowledge use. ‘The analysis shows that traffic effects will be null’ and ‘this is a pragmatic model, based on what is feasible,’ are valid arguments in the same discussion presented minutes after one another by the same person. The conception of ex ante analysis being simultaneously relative and absolute allows it to facilitate two types of discussions simultaneously. The analysis as something relative allows for a discussion on what Bertolini (2017) describes as a dilemma. A dilemma involves a choice between incommensurables, which are often difficult to quantify. What is more important, economic growth or reducing traffic nuisance? Who deserves better protection, people or birds?Footnote 3 These and more dilemmas occurred in discussions on what perspectives or indicators to use in policy processes, but they also served to relativize analysis outcomes. For example, displeasing results were attributed often to a narrow scope.

Where the relative side of analysis allows for a discussion on dilemmas, the absolute-seeming outcomes of an analysis facilitate a problem-solving approach. Problems are different from dilemmas in that they require finding a ‘best’ solution (Bertolini, 2017) instead of presenting a choice between incommensurables. Questions linked to problems include ‘How do we optimize traffic flow on this road?’ and ‘How do we build with a minimal impact on the environment?’.

In the early stages of the policy process, when the ex ante analysis is still being designed, there is ample time to discuss dilemmas. In all cases that we observed, stakeholders were asked to provide input for the design of the analysis by listing what they felt was important and should be taken into consideration. However, once the analysis starts, the analysis design becomes fixed. Now, the ex ante analysis has become a problem solver.Footnote 4 Its outcome—a cost–benefit ratio per alternative—does not necessarily facilitate a debate on dilemmas but instead allows a decision-making process to move forward. Reflecting on both relativity and absoluteness, ex ante analysis allowed two discussions to run simultaneously. The existence of two simultaneous discussions led to tensions. Where actors interested in completing the project wanted the process to move from optimizing the study to choosing a project alternative to be built, those critical of the project or a particular study wanted to keep discussing the scope of the study. This sometimes led to situations in internal meetings where people were forced to make decisions based on analyses whose scope they did not agree on.

To sum up, ex ante analysis appears to have built-in ambiguity. This is because the methods used allow the analysis to be relativized, but the way outcomes are presented simultaneously gives the analysis a sense of absoluteness. This ambiguousness allows analyses to reflect perspectives that present them as a solid ground for decision-making, such as linear perspectives and perspectives that emphasize that the use of analysis in policy making depends on one’s interpretation of the content of the analysis. The built-in ambiguity of ex ante analysis is, in other words, the first characteristic that enables it to be a chameleon in the policy process.

Split-up responsibility

A second characteristic of ex ante analysis that allows it to reflect many perspectives of knowledge use at once is that it does not appear to fully fall under anyone person’s responsibility. During our observations, we witnessed the conception of several analyses. Usually, this involved a group of civil servants standing around a flipchart and using thick, colored markers to write down things they believe should be in the analysis. This input was to be considered by the analysts who conduct the actual ex ante analysis. Despite their essential role in the construction of the ex ante analysis, analysts did not feel that they had ownership of the analysis they were conducting. Instead, analysts tended to portray themselves as people who merely execute. Oftentimes, they got their standards and input for models elsewhere. For instance, they often based their work on models developed by the department of transport. These models can be found in books and manuals, which are on the websites of the Flemish and Dutch governments. In Flanders, these standards have been provided by consultants. In the Netherlands, they come from a combination of government inhouse experts and private sector consultants. They reside in different places and are informed by different sources (Departement Omgeving; Ministerie van Infrastructuur en Milieu, 2016, pp. 101–136). In the process of conducting an analysis, analysts also got input from other sources. They organized citizen consultations, interviewed important stakeholders, retrieved data from government agencies, proposed analytical frameworks to a project’s political steering committee, and asked what they called ‘experts’ to estimate the effects of planned interventions. The substantial number of parties influencing ex ante analysis makes it impossible to assign ownership of the analysis to a specific actor. This feature of ex ante analyses caused actors to generally experience the ownership of the analyses as shared. When analysts received questions about the analysis they were working on, they regularly redirected those questions to other parties. When asked questions about the reasoning behind certain models, they referred to the government agency providing these standards. Most often, though, analysts—and others working with them—referred to the analysis itself as a source of authority. It was not the analysts, but the analysis that ‘showed’ or ‘indicated’ something. Despite many people being involved in the construction of the analyses we observed, the outcome of an analysis appeared to be seen as a product of the analysis itself rather than something created or influenced by analysts. When looking at where our code ‘absolute certainty’ overlaps with our codes ‘conclusions based on outcomes’ and ‘discussing outcomes,’ we see how an analysis which actors construct themselves produces outcomes that are discussed as if they are inevitable or undisputable:

‘If we would not build this project, we would see an immediate decrease of container traffic’

‘Whilst the problems are not pressing as we speak, they will be in the future’

‘All alternatives come with some risk, but—and this is important—all risks are controllable’

a: ‘There are also people living in this area, you know! You do not think about that!’

b: ‘We do, but it has been proven that the effects on traffic are nil’.

While no one is fully in control of the outcome of an analysis, the analysis does have control over the decisions a government makes. The fact that no one seems to be completely in control of the analysis means that it is difficult to challenge, but also that there is no one to respond to challenges. This had an impact on the power balance in policy processes. When outcomes are presented as a product of analytical models and are discussed as if they are indisputable, that acts as a shield between an analysis’ critics and the analytical choices that produced these outcomes. This is demonstrated in the following scene, in which two members of a project management team have a discussion about the interpretation of the results of an analysis. One of the team members pushes for stricter management of a set of sub-projects, as they fear these projects might be underperforming based on an ex ante analysis. The other team member aims to challenge this position by critiquing the analysis:

‘But where does that [projected result] come from?’ they ask, appearing increasingly irritated, ‘Out of nowhere!’ The project manager does not appear to know. The answer that they keep giving throughout the rest of the discussion, is that the outcome is just something that comes, ‘from under the bonnet’ of the analysis.

This scene demonstrates how, by accepting outcomes as a given, one also accepts the inputs of an analysis as a given. Consequently, actors wanting to criticize these inputs have to make an extra effort. Actors responsible for the inputs, on the other hand, are obscured behind the agency that an analysis exudes.

At the same time, the agency of ex ante analysis also allowed it to be used more flexibly. The fact that there is no ultimate authority dictating what is and is not a correct interpretation of an ex ante analysis allows ex ante analysis to be used and interpreted in contradictory ways. Because there is no definitive interpretation, people are relatively free to use ex ante analysis how they want. This freedom, which is a result of the way ex ante analysis is structured, constitutes its second chameleonic quality.

Trust

In the previous section, we explained that even though analyses are recognized as being dependant on assumptions that are openly being discussed, they do convey a sense of authority. After reading these sections, one might still have questions about the popularity of ex ante analysis. An analysis with built-in ambiguity for which no one appears to be fully responsible does not necessarily invoke a sense of trust. Nevertheless, in our observations we did not encountered anyone who dismissed the idea of conducting analyses altogether. Ex ante analysis appears to be trusted, despite the ambiguity surrounding it. In this section, we will explore what drives trust in ex ante analysis.

In our observations, one of the things that both opponents and proponents of a project asked for most when discussing an analysis was more detail. This points toward a first important source of trust: detail. Many of the field notes about people asking for more detail describe scenes in which people were worried that the analysts had overlooked something. Even in the early stages of a decision-making process, where the aim of the analyses is to provide information on only those factors that have been labeled as being of key importance, people asked questions such as ‘what will be the effect on village centers if the project forces farmers to relocate, and they drive their tractors along new routes?’ or ‘what will be the specific effects on fish living in the sunlight zone,Footnote 5 as opposed to just the general effects on the water as a whole?’ It appears that the more detailed questions an analysis could answer, the more people were inclined to accept its authority.

A second and related criterion is that actors expect analyses to possess a certain level of complexity. Things that are complex appear to be good. In one of the cases we studied, a project management team awarded a contract to a party partly because they had included a detailed plan of the analysis process. In another case, the chair of a public meeting called it positive that a nautical simulation was done by two computers, running ‘8 h a day, so it should be fine.’ In a third case, a critical expert invited to reflect on the process repeatedly suggested using a more complex model after several people had pointed out that this model was overly complex and did not fit the aim of the analysis. Citizens expressed their trust in ex ante analysis by stating that coming up with solutions for the policy problem at hand was ‘too complex for me, but you probably have all sorts of studies and analyses about that.’

A third criterion for trust is realism. In one case we studied, it was clear that actors expected an analysis to correspond to personal experiences for them to accept its outcomes. In this case, the traffic effects study showed no significant increases in traffic as a consequence of any of the alternatives that were being considered. In the eleven months we observed meetings for this case, no analysis was able to convince people who lived or own a business in the vicinity of the project that this outcome was realistic. In this case, ‘realistic’ meant one of two things: either the outcome has recognizability or corresponds with peoples’ sense of logic. Something is deemed realistic if a person can relate it to their own life. This is illustrated by the following statement by a citizen attending an information session:

He responds resolutely: ‘whoever doubts the relationship between traffic jams on the [riverbank], and whatever is happening in the ports, underappreciates reality!’

And by that same person, half a year later, making the following claims:

[H]e lives on one of the banks of the river impacted by the project and has recently experienced a company moving from one bank to the other. That really did lead to a substantial increase in traffic, so claims that he ‘really has trouble believing that moving and building new things will have no effects’. He calls for ‘creative thinking’ about solutions to fit more functions on the existing terrains. ‘That,’ he says, sounding a bit annoyed, ‘should be the priority’.

For something to seem realistic, the outcome must correspond with peoples’ sense of logic. An outcome must adhere to certain rules, such as ‘a transport alternative that requires frequently filtering in and out is not a viable alternative.’ This is the realism two civil servants in the scene below refer to:

The two representatives are standing near one of the pillars in the meeting room, sipping their cups of coffee. ‘Those self-driving busses’, one says to the other, ‘are totally without a chance. They’d need their own lane, and if they wouldn’t get that, imagine the number of times they’d have to filter in and out. . .’

Finally, people appeared to trust analyses because they could not be linked explicitly to (political) preferences of individual participants in the policy process, as these quotes from our field notes illustrate:

‘The question is whether or not that comes from the analysis, or whether [that preference] is wishful thinking’ (member of a project management team).

I’m genuinely completely surprised by this suddenly appearing alternative. The government told us they wanted to handle things objectively, and now they suddenly add this alternative’ (member of an action group).

‘I mean, the experts that did the study have been selected by the government; we don’t have anything to say about that. Why were we not invited to think about these matters’ (member of an action group).

In one way or another, these three process participants distinguished between that which is a personal preference, and that what is in the analysis. The member of the project management team distinguishes between the sensible analysis outcome, and ‘wishful thinking’ as the opposite of that. The members of the action groups distinguish between the objective analysis and the preference of government as not objective. In the last example, they voice their distrust in a study based on who selected the experts. Taken together, these examples illustrate how trust in an analysis appeared to be high as long as the analysis could not be linked to personal preferences.

What stands out in this overview of criteria for trust is that the content of an analysis is not one of the trust-enforcing factors. Rather, it seems that people were comfortable acting on the basis of an analysis which they did not understand, as long as it fulfilled the aforementioned criteria for trust. A second important notion with regard to trust, is that people appeared to universally accept the presence of ex ante analysis. Analyses were heavily debated, but no one ever proposed to stop conducting analyses altogether. In our cases, the trust-enforcing characteristics of the analyses processes appeared to outweigh any characteristics that might cause a decrease in trust, enabling the ex ante analysis to exhibit its chameleonic qualities.

Conclusion and discussion

This article started with the observation that both scientific research and policy practice contain a wide variety of theories on knowledge use. In this article, we study ex ante analysis as a popular form of knowledge to find out how it is possible that one instrument—an ex ante analysis—can reflect all these theories of use simultaneously. We find that when observing policy processes over a longer amount of time, one can indeed expect to see many a theory of knowledge use in action. This is partially explained by the fact that some theoretical perspectives of knowledge use are complementary. For instance, both linear and nonlinear perspectives can feature in a power game described by a critical perspective. In addition, a policy process often features a wide array of actors, which operate in ever-changing contexts and use knowledge in different ways throughout the process. Sometimes, a sub-decision is directly determined by a piece of knowledge; sometimes a nonlinear perspective offers a better explanation of a policy outcome.

However, what is of interest here, is that the different perspectives on knowledge use appearing throughout a policy process, apply to a piece of knowledge which remains stable in terms of the way in which it is constructed and conducted. This caused us to turn our attention to knowledge itself as a research object, to see if it has certain characteristics which allow it to reflect all these different perspectives on its use. In the case of ex ante analysis, we found that much like a chameleon, ex ante analysis changes appearance depending on its environment. As such, it can simultaneously facilitate several different perspectives of use, even if these are contradictory. Ex ante analyses can accommodate many types of use at the same time because of two properties: first, the analysis is both something relative and something absolute at the same time. On the one hand, an analysis is dependent on many assumptions and provides a limited representation of the outside world. This sparks debate about how different trade-offs should be reflected in the analysis. On the other hand, an analysis provides absolute certainty because the presentation of its outcomes eliminates all the nuances that are discussed during the construction of the analyses. As such, an analysis can be used in a linear fashion (the analysis says a, so we do a), as well as more nuanced ways such as Stevens’ (2011) story-crafting or Weiss’ (1979) tactical model. This observation also confirms Boswell’s (2009) paradox of knowledge use, namely that policy makers are draw heavily on research in debates (as if outcomes were absolute) yet are hesitant to be overly reliant on it (relativizing these same outcomes).

The second property which allows an ex ante analysis to display its chameleonic behavior, is that an analysis enjoys a certain independent agency. Even though people were themselves involved in the construction of an ex ante analysis, they discussed analysis outcomes as a product of the analysis, rather than analysts. This was largely due to the fact that the responsibility over an ex ante analysis was split up over different actors and institutions, so that no-one in the policy process was completely responsible. As a result, analysis could be used and interpreted in many different ways, as there was no authority determining what the true interpretation or right use of an analysis would be.

Despite its chameleonic properties, people trust ex ante analysis. Factors that enforce trust are detail and complexity, realism (i.e., people can relate outcomes to their own experiences and sense of logic), and the idea that analyses provide an alternative to the undesirably large influence of political values over facts. These trust-enforcing characteristics appear to be enough to overcome any potential decreases in trust caused by other characteristics, making them an essential enabler of ex ante analysis’ chameleonic qualities.

This study’s primary contribution to the literature is that it suggests a shift in focus from people’s behavior to pieces of knowledge as an object of study. The majority of studies on knowledge use focus on how actors in policy processes use and interpret knowledge. The theoretical framework of this article shows that this has led to a broad spectrum of perspectives on knowledge use, one not necessarily truer than the other. This study suggests that to escape adding to this ever-expanding set of descriptions of actors’ behavior, it might be more fruitful to focus on how these behaviors come to be as a result of the interaction between process actors and specific properties of, for example, an ex ante analysis.

Furthermore, our study gives rise to three more specific questions. First, a legitimacy question: if ex ante analysis is discussed as if they are free from human interference and do not include any political values, but are in fact a human product, what legitimizes their impact on policy processes? Second, even though our research used cases from two very different administrative cultures, there were similarities across all cases which might have impacted our results. For example, all ex ante analyses we encountered were conducted by contracted private parties, and the topics studied were similar across all cases. Future research could show whether forms of policy knowledge with different properties would bring about different behavior in process participants. The third and final question is this: what effects do the chameleonic qualities of ex ante analysis have on policy conflict? We observed that people trust ex ante analysis more than they trust other people, who they often suspected to act on a personal agenda as opposed to what is ‘objective.’ Simultaneously, ex ante analysis structures discussions by deciding which arguments can and cannot be included in different stages of a project (Jasanoff, 1990; Parsons, 2002), and its multi-interpretable character brings up associations with concepts such as ‘fact-free’ or ‘post-truth’ politics (e.g., D’Ancona, 2017). In other words, ex ante analysis facilitates debate, but also closes off avenues for discussion and potentially confuses policy debates. Whether ex ante analysis leads to more productive policy processes remains an open question.