Elsevier

Poetics

Volume 82, October 2020, 101475
Poetics

The fuzzy middle. Uncertainty, indifference, and disagreement in the evaluation of contemporary art music

https://doi.org/10.1016/j.poetic.2020.101475Get rights and content

Highlights

  • The middle of a ranked list is evaluated differently from the top and the bottom.

  • In this “fuzzy middle”, outcomes are unpredictable and often arbitrary.

  • Lack of recall, indifference, and disagreement affect outcomes of mid-level players.

  • The role that emotional reactions should play in the evaluation is uncertain.

  • Criteria are developed ad hoc and on a case by case basis in the “fuzzy middle”.

Abstract

This article looks at peer review in the “pure” pole (Bourdieu 1996; Dubois and François, 2013) of the artistic field, in contemporary art music. Based on observation of peer review panels in the United States and interviews with panelists in the US and France, I look specifically at the dynamics of evaluation in the upper middle of ranked lists – what I call the “fuzzy middle”. In this part of ranked lists, outcomes are unclear and often arbitrary. Two factors primarily affect outcomes: 1) a lack of recall on the part of evaluators regarding the specifics of the objects being evaluated, which leads to an attitude of indifference, and 2) confusion as to how disputes over artistic quality should be resolved. I find that uncertainty is present especially regarding the role that emotional reactions to the music being evaluated should play. In addition, not all participants are willing or able to participate equally in the evaluation process (Lamont 2009), and the interpersonal dynamics of these panels therefore heavily affects the outcomes of the evaluation. What results is that objects in the “fuzzy middle” are evaluated collectively using criteria that are developed ad hoc and on a case by case basis, as a way to translate emotional reactions into objectifiable criteria. I do not find, however, that quality uncertainty (Karpik 2010; Menger 2014) plays a role in these evaluations: at least at an individual level, these evaluators do not express doubts about their evaluations of artistic quality.

Introduction

I really found this work very strong – I was really into her music, the samples really impressed me. I really liked the project.

There's not a lot of strong for or againsts really, it's just kind of there.

The work itself didn't grab me. The samples, you know, didn't stand out to me as something remarkable, though I recognize the craftsmanship and the effort, and the creativity and so forth.

(excerpts from panel debates, American organization, 2012)

How do we understand these statements, given that artistic fields are plagued with quality uncertainty (Karpik 2010; Menger 2014)? How are these evaluators able to make these statements with confidence, and what effect do these statements have on the fate of the applicants they are evaluating? This article seeks answers to precisely these questions. My research addresses the evaluation of contemporary art music1, and, specifically, how composers are evaluated by their peers when they apply for funding for their projects. I look at how evaluation takes place in two peer review settings, one in France, the other in the United States. Peer review, when it includes face-to-face discussions as these panels did, explicitly values the interpersonal influence of this setting, although, as we will see, many safeguards are in place to control and channel this influence.

When I was conducting this fieldwork, from 2011 to 2016, I noticed that the judgments I observed were largely decisive and quick – as has been observed in other, similar settings: an architectural competition (Kreiner 2012), editorial meetings of the fiction staff at a literary magazine (Merriman 2017), fashion producers choosing models (Godart and Mears 2009), or an admissions test for a design school (Strandvad 2014). Given the quick, intuitive character of the judgments I observed and which are described in the literature, I began to question the role that uncertainty plays in the day-to-day workings of these settings, where singularities (Karpik 2010) – despite being incommensurable – are regularly compared. This article teases out where uncertainty in these types of judgments arises and why it might appear. Based on empirical material from two cases, I will show that outcomes are significantly more unpredictable for candidates ranked toward the upper middle of a ranked list, and less so for those to be found at the top or bottom of such a list. This is the area I have chosen to call the “fuzzy middle”, as this is where outcomes are unclear. This focus on the upper middle, on this musical reserve army, if you will, is important as much of the literature on this type of evaluation looks at how success at the top can be explained. In the cases studied, the unpredictability in the “fuzzy middle” arose from disagreements and indifference within the peer review panels, and was not due to uncertainty regarding quality judgments. This means that the uncertainty present in these evaluative settings – in the form of unpredictable or arbitrary results – was the result of interpersonal dynamics of the collective work of these peer review panels, and not due to the intrinsic characteristics of the objects being evaluated.

This article is organized as follows: in the next sections, I will first (I.a.) give a definition of the term “contemporary art music” and the milieu to which this refers, followed by a discussion of the sociological literature on evaluation under conditions of uncertainty (I.b.). Then (I.c.), I will present in more detail the organizations I studied and the kind of data I was able to collect. Following this, I will look at some general features of the evaluative panels I studied (II.a.), and specifically how the upper middle of a ranked list was treated differently from the top and the bottom. This will lead me to look in detail at one specific case (II.b.), to understand better the underlying dynamics of evaluation in this milieu. A discussion (III) of these results will follow, and the article will close with a brief conclusion (IV) outlining the main contributions of this research and its applicability to other settings.

The fieldwork for this study was conducted in the world of contemporary art music. Though, as in any art world, determining what is part of an artistic genre is an object of considerable attention within the art world in question (Becker 1982, 36), the following general definition gives the context necessary to understand what is at issue in the present study:

[Contemporary art music comprises] pieces that [are] composed or preplanned reflectively, fixed in some sort of notation for a performer or creator to interpret or execute, and intended to be listened to by an attentive, informed, and critical audience. We might add that it is a style of music that traces its primary lineage back to the courts and churches of pre-Renaissance Europe, and although those courts and churches are today mostly long defunct or culturally marginal, contemporary art music maintains an important relationship with their modern-day descendants and the structures of production and listening that they represent (Rutherford-Johnson 2017, 3).

To situate this more precisely, some composers who have become more or less household names working in this genre include, historically, John Cage, Karlheinz Stockhausen, and Pierre Boulez – and for composers working currently, we might cite John Adams or Philip Glass. In terms of funding, government structures and non-profit organizations have primarily taken on the role once played by courts and churches, and the world of contemporary art music, both in France and the United States, is indeed highly subsidized. It is made up of a majority of actors who do not make a living from their artistic practice, as most composers and performers working in this domain have a “day job,” so to speak, as is the case in most artistic professions (Menger 2014, 124), often as teachers (Jeffri 2008).

However, it must be stated that “contemporary art music” is not the term that the organizations I studied use to describe the music they fund. I have chosen to use this term nonetheless, for two reasons. Firstly, to keep the organizations I studied anonymous, as this is a field in which idiosyncratic terminology tends to proliferate, and secondly, because this term is relevant in the sociological literature on art, where scholars have studied extensively how distinctions between art and craft (Becker 1982) or highbrow and lowbrow cultural production (Levine 1990) have emerged. This term is widely used in the field of musicology and other fields which study music2, and the use of this term indicates where this production is situated in terms of its cultural and symbolic legitimacy.

Much of the literature on value in cultural fields tries to understand why some individuals and works are more successful than others. Pierre-Michel Menger posits that small, underlying differences in talent become exaggerated in processes of cumulative advantage (Menger 2014), while John C. Huber argues for the importance of simply not giving up – tenacity (Huber 2000). What I will argue here is that these are plausible explanations for the success of the “winners”, but that the dynamics of the middle players are much less clear and can often be arbitrary. Given that these mid-level players represent a much larger population than the one at the top, it is important to understand how their work is evaluated in these fields.

Any process of evaluation, valuation, appreciation, appraisal… begins with attention (Hennion 2015). Attention can be focused in different ways. Importantly for this study, quantitative commensuration makes certain things visible and hides others (Espeland and Lom, 2015), and this “rigorous relativity […] prevents the expression of singularities” (Espeland and Lom, 2015), 25). Singularities are goods whose fundamental value is their uniqueness and no “objective hierarchy” can be used to evaluate them (Karpik 2010, 39). The present case, that of contemporary art music, is one where uniqueness is prized and the expression of the subjectivity of the composer in her music is of the essence. How, then, do we arbitrate between these unique expressions of individuals? Lucien Karpik states that the public presentation and marketing of singularities necessarily implies an “arbitrary selection of certain dimensions to the detriment of others” (Karpik 2010, 41). Thus, “all valuation hinges on relations of visibility” (Espeland and Lom, 2015, 35). What is hidden or ignored cannot be compared, and evaluation processes seek precisely to organize attention through the use of formal criteria or quantification, for example, in order to obtain results that are coherent with a certain ideology or other standard.

I am combining two contrasting bodies of work here. I use Bourdieu (1996) to understand the shape and features of the contemporary art music milieu, which fits into what he describes as the “pure” pole (Dubois and François, 2013) of the classical music field. This pole is one of restricted production, wherein immediate, popular success is shunned and the admiration of a restricted circle of peers is sought instead (Ibid.). Thus, the lay audience's opinion is disregarded, and it is the approval of peers working in this milieu that is necessary to achieve success as it is defined here. To this descriptive perspective, I add the work of Lucien Karpik (2010), who looks primarily at the point of view of the consumer unable to mobilize the expertise necessary to choose between singularities, and who therefore turns to judgment devices of various natures to make his decisions. Karpik's analysis does not distinguish between pure and commercial poles. Rather, he tries to understand in detailed ways how consumers overcome informational and cognitive deficits in order to choose between seemingly incomparable products. Works of contemporary art music would qualify as “singularities” in Karpik's theoretical analysis, in that they “must differentiate themselves by their degree of originality, which, in theory, renders them poorly comparable” (Menger 2014, 160). This means that the evaluators in these milieus are using techniques like those described by Karpik (2010) for the consumer faced with singular products. Evaluators also resort to judgment devices, which, for contemporary art music, might be educational background, interpersonal networks, or the judgment of festivals and orchestras. These individuals are working in a subfield of restricted production (Bourdieu 1996), which implies a certain level of expertise, but I will argue that they are making decisions like the humble consumers described by Karpik (2010).

I will argue in what follows that evaluation is first and foremost a question of categorical fit, in that the arguments that I encountered are less about quality than they are about whether or not the production in question fits the categories proposed by the organizations as it is understood by the evaluators. In addition, the categories themselves, as will be seen clearly in the cases presented, also emerge through evaluation. Even a category as taken for granted as “nature” is “dependent on the methods – legal, economic, and ecological – that [are] mobilized to account for it” (Fourcade, 2011). Depending on the evaluator's place in the field in question, her way of drawing disciplinary boundaries will change (Lamont, 2009) – and the judgments made on the manuscripts or works to be evaluated are also implicit judgments of the work of one's colleagues sitting around the same table (Hirschauer 2010, 92). One aspect of these judgments is cultural matching (Childress and Nault 2019; Rivera, 2012): applicants that have similar backgrounds and artistic ambitions to the evaluators are viewed more favorably.

But whence the uncertainty? Whether in the market for contemporary art (Velthuis, 2003, 193) or in the careers of artists more generally (Menger, 2014), value is seen to be radically uncertain for artistic production. Lucien Karpik sees incommensurability as a primary source of uncertainty when singularities are under the microscope (Karpik, 2010). This means, simply put, that when uniqueness is the leading evaluative criteria, comparison is inherently difficult. In Menger's analysis, artists cannot know the value of their work until they enter this field of competition, and participating in comparative tournaments is seen as a demand for information on their own relative level of talent (Menger, 2014). The value of creative labor can only be known from comparative evaluation (Ibid., 118), and can only be measured in relative terms (Ibid., 179). These tournaments act to lessen uncertainty as to whose work is most valuable, by magnifying minute differences in talent in a process of cumulative advantage (Menger, 2014), with talent being defined as “the quality gradient attributed to the individual artist through these relative comparisons” (Ibid., 180). This definition of talent assumes that there are intrinsic differences in ability, but that they are not fully observable and that we infer qualities from the attention others give to an individual (Ibid., 230-231). The present article will look closely at the inner workings of such comparative tournaments and the role that talent and uncertainty may or may not play in these judgments.

Two case studies inform this text, one conducted in the USA, the other in France. The US case study investigated a semi-public, non-profit organization whose sole purpose is to support contemporary art music, whereas the French organization is a governmental body which funds and legislates cultural matters. Within these organizations, I looked exclusively at funding schemes for composers of contemporary art music and the way their applications for funding were evaluated. Some of the data were derived from direct observation, in 2012 and 2013, of two peer-review panels in the American organization. I conducted semi-structured interviews with the evaluators in the days following these panels. In addition to these interviews, I conducted a series of interviews with panelists from the organization between 2011 and 2016, as well as interviews with all members of staff of the organization. The French organization did not allow me to directly observe jury discussions, and thus I gathered my data via interviews with the participants of the evaluation committees and staff members of the organization in question over the years 2012 to 2016. These interviews were enriched by a series of interviews with other members of the contemporary art music world in both countries, including composers, performers, commissioners, critics, and others. A complete list of interviews is included in Appendix 1.

The American organization convened juries of 3 composers who met over the course of 2-3 hours. The composers were recruited from the field of previous recipients of the grant being distributed, with an eye for diversity: at least one woman was recruited of the three, and the composers were supposed to represent different aesthetics within the world of contemporary art music, and geographical diversity was also important. The grants awarded by the juries I observed ranged from a few hundred dollars to a maximum of 5,000 USD, whereas the French grants ranged from 5,500 to 38,500 EUR. The French organization also had much larger panels – 14 members, with six being composers, two musicians, two concert or festival organizers, three representatives from the governmental organization, and a chair (all members had voting rights). This panel met for five full days over the course of one week. Similar to the American organization, the primary condition for being recruited was to have been a recipient of the grant in question, and stylistic diversity was also important. Starting in 2014, the French panels were required to be 50% women. Prior to that, in the years 2002-2013, women represented between 10% to 30% of the panel.

I transcribed and inductively coded my interviews using Atlas.ti in order to reveal the different evaluative schemas used by the evaluators. Justifications for quality assessments were coded in detail, such that operative judgment criteria could be clarified. I also transcribed the panel discussions I was able to observe directly and inductively coded them for operative criteria, as well as for argumentation strategies and patterns of interaction and turn-taking, following conversation analysis methods (Have, 1999)3.

These cases are meant to be compared as two most similar cases: two organizations working to find a solution to the same problem, namely, how to evaluate contemporary art music. These cases are complementary and additive, in the sense that when they are taken together, we gain a greater understanding of how evaluation of artistic production in the pure pole takes place, and glean insight into cultural differences in this supposedly international milieu. Because of the difference in access I had to these two organizations, however, I present them here with a focus on the American organization, with the French organization serving primarily as a foil to understand how things could be done differently. The organizations are different in their funding structures, in that the American organization is a semi-public organization, whereas the French program is purely government funded. For the American organization, this implies conflicting priorities of different donors, and the evaluation process is set up to resolve these conflicts (see Fryberger, Annelies (in press)).

When this fieldwork was done, both the French and American organizations convened their juries for face-to-face meetings (online in the US, in person in France) in order to evaluate the applications they had received. For the funding schemas studied here, the American organization required their jury members (3 per jury) to evaluate all the applications (around 70, of which approximately 30 were funded) prior to the meeting and to provide a numerical score (from 1 to 6, 6 being the best) on three criteria (see Table 1). As such, these panels were similar to those studied by Lamont (2009), in that they involved the creation of a rough ranking which was then refined in discussion. The French organization asked each of its panelists (14 total) to evaluate a subset (10 to 12, typically) of the applications received (between 130 and 220, of which 50-60 were funded) and to act as a rapporteur for these applications during the meeting. Every application was presented with an opportunity for discussion, whereas in the much shorter American meeting, only a selection of applications was discussed. In the French organization, there was a secret and binary vote – each jury member had to check on their ballot if each of three criteria (see Table 1) was met or not, and the ranked list thus obtained was not discussed by the panel. The American organization based its final, ranked list on the discussion results – the final list was approved by the panelists at the end of the meeting.

The organizations asked their jury members to evaluate applications based on the criteria presented in Table 1. We will look at how these criteria were used in the judgment process, but, for now, it is important to understand the meaning given to these criteria by the organizations themselves. In the American organization, only one criterion dealt directly with the artistic merit of the proposed project and the composer – that would be the first, that of “artistry.” The other criteria were more technical and intended to be more objective: “impact” designated the impact that the proposed project would have on the composer's career, while “appropriateness” referred to the feasibility of the project and an assessment of the actual costs of the project compared to the sum requested in the application. The latter criterion was meant specifically to evaluate the project's budget, as the organization required that costs be correctly documented and explained in the application. In the French organization, all three criteria dealt in some way with the artistic merit of the proposed project and the composer. The first criterion addressed the composer herself and her previous work – the proposed project was not to be taken into account. The second looked at the quality of the proposed project, and the third the performance conditions. Inherent to the third criterion was an assessment of the musician or ensemble who would premiere the piece, while “career of the work” referred to whether or not subsequent performances were scheduled.

Section snippets

The fuzzy middle

The evaluation panels I observed in the American organization had similar features in terms of the way results were distributed: they both featured a silent consensus regarding the applications ranked highest and lowest, with the most discussion being about the applications that ended up in the upper middle. It is easier to understand this fuzzy middle – the “gray zone” (Composer 13, France) – if we look closely at the jury discussions observed in the American organization in 2012 and 2013. In

Discussion

A first, important point: the organizations studied here are not trying to manage uncertainty by using a peer review system – if anything, this system introduces uncertainty, in the form of arbitrary results, into their decision-making processes. In the American case, by only recruiting evaluators from the highbrow end of music production, and never composers working in more popular genres, the organization can position itself in the “pure” pole without stating this goal explicitly. The

Conclusion

This research was conducted on peer review in contemporary art music, on systems that use rankings to make funding decisions. The conclusions discussed here likely apply to other similar situations, settings where artists are applying for grants or where panels are tasked with arbitrating acquisitions or other, similar settings in the “pure” pole (Bourdieu, 1996) of the artistic field. I would hesitate to apply these findings to more popular artistic fields, as the desire to appeal to a mass

Funding

Research funded by a Ph.D. contract from the CRAL (Centre de recherches sur les arts et le langage) of the EHESS (Ecole de hautes études en sciences sociales), Paris, France.

Declarations of interest

None.

Data statement

A list of interviews conducted for this research, as well as transcripts of the vast majority of them, is available from the author on request. These documents accompanied her Ph.D. thesis when it was submitted.

Acknowledgements

The author wishes to acknowledge Esteban Buch and Nicolas Donin for their support for this research, as well as the participants in Hyacinthe Ravet's seminar at the Sorbonne on the sociology of music and participants in the Spring School “Organized Creativity” at the Freie Universität Berlin, especially David Stark and Tobias Theel, for their comments, and Jens Beckert for his reading of previous versions of this manuscript.

Annelies Fryberger is a sociologist of music. She obtained her PhD in 2016 at the EHESS (Paris, France) where she was a member of the Center for Research on Arts and Language (CRAL-EHESS) and the research team “Analysis of Musical Practices” at the IRCAM. Her research has been published in Figures de l'art, N. XXX (2015), Circuit. Musiques contemporaines (2016), Proteus (2018), among others. She was previously a postdoctoral researcher at the University of Hannover (funded by the DAAD) and the

Bibliography (33)

  • Marion Fourcade

    Cents and sensibility: Economic valuation and the nature of "Nature"

    American Journal of Sociology

    (2011)
  • Marc Gidal

    Contemporary Latin American composers of art music in the United States: Cosmopolitans navigating multiculturalism and universalism

    Latin American Music Review

    (2010)
  • Frédéric C. Godart et al.

    How do cultural producers make creative decisions? Lessons from the catwalk

    Social Forces

    (2009)
  • Paul ten Have

    Doing conversation analysis: A practical guide

    (1999)
  • Antoine Hennion

    Paying Attention: What is Tasting Wine About?

  • Stefan Hirschauer

    Editorial judgments: A praxeology of 'Voting' in peer review

    Social Studies of Science

    (2010)
  • Annelies Fryberger is a sociologist of music. She obtained her PhD in 2016 at the EHESS (Paris, France) where she was a member of the Center for Research on Arts and Language (CRAL-EHESS) and the research team “Analysis of Musical Practices” at the IRCAM. Her research has been published in Figures de l'art, N. XXX (2015), Circuit. Musiques contemporaines (2016), Proteus (2018), among others. She was previously a postdoctoral researcher at the University of Hannover (funded by the DAAD) and the Labex CAP (IRCAM/Pompidou Center), and is currently a visiting scholar at the New School.

    View full text