Annotating Speaker Stance in Discourse: The Brexit Blog Corpus

1 Introduction

Communication between humans is never completely neutral in the sense that no particular perspective or selection of information is employed. Human interaction is always view-pointed in one way or another, and meanings of lexical items are dynamic and adaptable in relation to the situational and linguistic context where they are used (Langacker 1987; Talmy 2000; Croft and Cruse 2004; Gärdenfors 2014a, 2014b; Dancygier and Sweetser 2012; Paradis 2015). Speakers take stance when interacting with other people. They make assessments and they position themselves in relation to other interlocutors to mark their standpoint (Englebretson 2007).

Speaker contributions come with stances expressed in a range of different ways that may convey more than one stance in different contexts, and there may be more than one stance expression in the same sentence or utterance. There are cases of expressions of stance such as might, definitely, I am sure that that are treated as expressions of stance in the literature, but there are also types of constructions that might or might not be included in the category of speaker stance. For instance, It’s about time you sold your car where the sequence starting it’s about time + past tense of sell expresses a rather direct and tactless recommendation to somebody else about what the speaker thinks is beneficial for the addressee. Utterances of this kind are pervasive in language use and make the study of stance in real communication intriguing, but also methodologically challenging. A challenge that lies ahead of us concerns how we can go about identifying the meanings and the forms of stance in order to be able to train a computational model to automatically identify stance in discourse. For that purpose, we need a solid theoretical ground to start from, consisting of robust criteria for detection and annotation with acceptable inter-coder reliability scores.

A comprehensive cognitive-functional framework consisting of ten notional categories for annotating stance was set up, where the basic units of analysis are utterances. The term utterance in the present study is defined as the chunk between full stops. The utterances were holistically analyzed to determine whether they expressed speaker stance or not by two expert analysts, who also identified what type(s) of stance were expressed in each utterance. The main principle for assigning speaker stance to an utterance was that stance-taking should be identifiable through chunks of form-meaning pairings of varying size in language, which we refer to as constructions (Fillmore et al. 1988; Langacker 1987; Goldberg 1995, 2006; Croft 2001). This procedure contrasts with a great deal of work both in quantitative corpus linguistics (Biber 2006) and sentiment analysis in computational linguistics (Pang and Lee 2008), where the starting point is a preconceived list of words that the researcher assume generally express a given sentiment type. In contrast to these works, we offer an utterance-based approach to the analysis of speaker stance in communication based on the identification of constructions that actually express stance on the occasion of use. The prospective usefulness of this study, couched in a usage-based theoretical approach to meaning in language (Paradis 2015), is automatic identification beyond mere form, i.e. lists of words.

In order to come to grips with how speaker stance is expressed in language, and how different stance categories interact in discourse, ten core speaker stance categories have been defined on the basis of the rich literature on stance in language (for a comprehensive, annotated stance bibliography, see Glynn and Sjölin 2015). We compiled a corpus of social media text from blogs, the Brexit Blog Corpus (BBC), consisting of blog posts commenting on political issues related to the 2016 UK referendum. BBC was annotated by two annotators, and the annotation results are evaluated and discussed.

The paper is organized as follows. Section 2 presents an overview of the basic stance-taking concepts and studies in the field. In Section 3, the ten categories of stance are defined and exemplified. In Section 4, the BBC is introduced. It is described, and the annotation process and the results are presented. Section 5 evaluates the annotation results and discusses the cases of multiple stance occurrences in the corpus data. Finally, Section 6 is a summary of the findings and the implication of this study.

2 Background

In this section, an overview of different concepts that are used in the literature to talk about stance and studies from different academic traditions, theoretical as well as computational approaches, are considered and compared.

2.1 What is speaker stance?

Stance-taking in verbal communication among people can be described as the expression of the speaker’s assessment of an object, an event or a proposition vis-á-vis his or her interlocutors. Stance is defined as the way speakers position themselves in relation to their own or other people’s beliefs, opinions and statements about things or ideas in ongoing communicative interaction with other speakers. Speaker stance is firmly grounded in the speech situation and as such, stance-taking is crucial for the social construction of meaning in different discourses. In this study, we follow Du Bois’ (2007) definition of stance.

One of the most important things we do with words is to take a stance. Stance has the power to assign value to objects of interest, to position social actors with respect to those objects, to calibrate alignment between stance takers and to invoke systems of sociocultural value.

(Du Bois 2007: 139)

Rather than providing a catalogue of stance expressions, Du Bois aims at a general understanding of the phenomenon as such. For that purpose, he finds it is necessary to pinpoint the foundational principles of taking stance and negotiating meanings (Figure 1).

Figure 1:

The stance act represented in the form of a stance triangle in Du Bois (2007: 163).

As Figure 1 shows, evaluation, positioning and alignment are three different aspects of a single stance act, where each aspect is distinguishable from the others through the consequences it has in the act. In the stance act, the stance taker evaluates an object, thereby positions himself or herself, and aligns with other subject(s). Du Bois’ way of representing stance is important for our understanding of the phenomenon as such. However, to complete the picture we also add some more specifications about its nature.

1. Stance is a psychological state involving speaker beliefs, evaluative ability and attitudes.

2. Stance-taking is the performance by humans in communication, actions taken by speakers to express their beliefs, evaluations and attitudes toward (i) objects, scenes and events, and (ii) toward propositions, and speakers’ viewpoints on what is talked about.

3. Expressions of stance are constructions that are used to take stance and express meanings relating to speakers’ beliefs, evaluations and attitudes.

Even though stance as a psychological state, as in (a) is important for our understanding of what stance is, it is beyond the scope of this study and therefore not addressed at all. Rather, it is considered a psychological prerequisite for this investigation. Stance-taking (b) and expressions of stance (c), on the other hand, are both ingredients of stance constructions and important for our analysis. They are the meaning side and the form side of what we find in the utterances.

Formally, stance markers are notoriously difficult to specify in advance because unlike some other categories, they are not confined to traditional areas of grammar, morphology or vocabulary, but to all of these as well as to longer chunks or even whole sentences. As has already been mentioned, some stance expressions are easily identifiable as such (perhaps, surely, must, I don’t know) because they are elements that always and unambiguously express speakers’ assessments. There are other constructions, however, that may not be thought of as stance markers in the first place since they are parts of lexical items such as -able/ible as in doable, drinkable, possible, but all of them come with a possibility judgement. The utterances in (1)–(4) below are all examples of advice of some sort, which can be expressed in a number of different ways. From the point of view of the words used to express the recommendations, some such utterances contain words that express a ‘request’ or ‘recommendation’, such as I suggest (1) and (2) and I recommend in (4), while (3) has no clear indication in terms of the individual words. In addition, the use of the past tense in (1) and (3) is an indication of distancing or toning down the speaker’s accountability.

(1)

I would suggest that you left your house before the end of this month.

(2)

I suggest that you leave your house before the end of the month.

(3)

This wine should drink well within the next couple of years.

(4)

I recommend that you drink this wine within the next couple of years.

The use of the past tense, as with would in (1), evokes a distancing speaker stance. The speaker stands back and makes a tentative and polite recommendation to the addressee, while the opposite holds true of the speaker stance in (2), which is direct and therefore also much more of an urgent and potentially rude order. In (3), the speaker hides behind the wine. As the subject of a transitive verb in a middle construction, the wine assumes animate powers with the effect of backgrounding speaker accountability as an assessor. In addition, the use of should in the construction adds speaker tentativeness and potential uncertainty on behalf of the speaker. The active first-person construction in (4), on the other hand, forces the speaker to stand up for his or her assessments (Paradis 2009; Hommerberg and Paradis 2014). As our approach to stance-taking is cognitive-functional and usage-based, we are interested in how speaker stance is conveyed in real utterances. For this reason, we explore stance from the point of view of how we interpret the meaning and the illocutionary force of the utterances in our corpus.

3 Stance categories

For the purpose of this study, we have identified a broad framework of ten notional stance categories from the literature reviewed in Section 2. In alphabetical order, the categories are listed as follows: agreement/disagreement, certainty, contrariety, hypotheticality, necessity, prediction, source of knowledge, tact/rudeness, uncertainty and volition, as shown in Table 2. This list comprises a wide spectrum with the most important stance-related concepts according to the research described in Section 2.

In Table 2, each category is followed by a brief description of the type of stance-taking associated with each notional category. Examples are given in the third column. The stance-taking elements are constructions of varying length. For instance, Don’t you think it might be a good idea to postpone the meeting until tomorrow is formed in a tactful and polite way while the word wanted in We wanted him to sell the house is solely responsible for the volitional reading. As already pointed out, stance constructions include individual stance-marking items such as must, could, possibly as well as longer stretches such as It’s about time you sold your car where the sequence starting it’s about time + past tense of sell are indicators of an interpretation that expresses a rather direct and rather tactless recommendation to somebody else about what the speaker thinks is beneficial for the addressee.

The descriptions of the stance categories in Table 2 were used as instructions for the annotators. The categories are not mutually exclusive but co-occur in the utterances. For instance, the category source of knowledge includes evidence from the five senses (sight, touch, hear, taste, smell) as well as expressions referring to inferential reasoning such as it seems. The coding for an utterance with it seems is not only an expression of knowledge source but also of uncertainty and maybe also other stances depending on the utterance as a whole, as we will see in Section 4. Furthermore, some categories encompass two properties. For instance, hypothetical constructions per definition involve a prediction. A decision was, however, made not to place them in two categories, but to code them as hypotheticality only since such utterance will always also be prediction. Two of our categories comprise opposing notions within the same meaning-function domain: agreement and disagreement are concerned with alignment versus disalignment with the addressee at the functional level, while tact and rudeness are contrastive notions on the dimension of politeness. In the case of contrariety and necessity, the naming of the categories contains two similar representations. Contrariety involves both concession and contrariness, which are both contrastive construals (Paradis and Willners 2011; Jones et al. 2012), and necessity includes need and requirement, e.g. obligation, requests for the benefit of the speaker and recommendations which are issued by the speaker for the benefit of the addressee (Paradis 2009).

4 The Brexit Blog Corpus

In this section, we describe the design of the BBC and how the collection of the data was carried out. The rationale for the compilation of the corpus is summarized in (i)–(iv).

1. This social media text type was expected to contain subjective language and thus stance-taking information.

2. The BBC serves as the benchmark to test the validity, applicability and usage patterns of the ten stance categories.

3. The BBC enables us to carry out further computational research on stance in order to arrive at a more integrated and plausible analysis of speakers’ positioning, alignment and evaluation in online social media discussions.

4. The annotated gold standard corpus is a linguistic resource, which will be publicly available to the research community for further study and analysis through SND. ^[3]

In Section 4.1, we describe the data collection process and in Section 4.2 the annotation.

4.1 Corpus description and annotation process

The BBC is a collection of texts from blog sources. In line with principles governing traditional corpora (Gilquin and Gries 2009), it consists of a time-bound data set of networked assemblages of streaming data (Zappavigna 2012). In the literature, blogs are defined as a distinct text type differing from social media texts from other sources in terms of structure and size (Baldwin et al. 2013; Myers and Hamilton 2015; Berger et al. 2015). The blog post texts can be considered as a text type that shares many characteristics with impromptu spoken conversation in that the contributions are relatively spontaneous and very often not supervised, and not revised or according to the norms of formal writing. However, blogging suffers less than microblogging, e.g. Twitter posts, from features that impose difficulties in analytical tasks such as non-standard orthography, omitted characters and other spelling issues, special characters and XML tags.

The corpus texts are thematically related to the 2016 UK referendum concerning whether the UK should remain members of the European Union or not. The texts were extracted from the Internet from June to August 2015. With the Gavagai API, ^[4] the texts were detected using seed words such as Brexit, EU referendum, etc. We retrieved and filtered only URLs ending in wordpress.com, blogger.com, blogspot.*. Only entries described as blogs in English were selected. Each document was split into sentential utterances, from which 2,200 utterances were randomly selected. The final size of the corpus is 1,682 utterances, 35,492 words (169,762 characters without spaces). Each utterance contains from 3 to 40 words with a mean length of 21 words.

For the data annotation process the Active Learning and Visual Analytics (ALVA) system implemented by Kucher et al. (2016a, 2017) was used (see Figure 2). At the top of the annotation interface page, the utterances to be annotated appear in a box one at a time and beneath the utterance box are the stance categories and three options: (i) Submit annotation, (ii) Mark as neutral and (iii) Mark as irrelevant. The first option was used in cases where at least one of the stance categories is present. The second one when the utterance did not contain any explicitly worded stance-taking expressions. The third option was used for incomplete utterances and boilerplate text chunks such as Posted by [name], i.e. where no statement, question, promise or order is issued. As mentioned above, annotators could attribute more than one category in the same utterance as in (5)–(7).

Figure 2:

Annotating with ALVA: the annotator’s decision about the utterance However, since we don’t know much about her ideology, there’s just as large a chance that she won’t and will be a repeat to Ed Miliband too. is prediction and uncertainty.

(5)

Different political contexts, I know, but the core principle of the Eurozone that the bondholders must be paid every cent due to them – will be defended to the hilt and regardless of the human consequences. → agreement, contrariety, prediction

(6)

Regrettably I believe that Farage is going to prove to be a negative influence that may even guarantee an unsuccessful outcome. → uncertainty, prediction

(7)

Once again, this is not the talk of doom-mongers, but ‘respected’ institutions everywhere. → source of knowledge, contrariety

Neutral is a problematic notion, which we make use of in a simplistic and practical way. Utterances were categorized as neutral in cases where none of our ten stance categories was identified, like in the examples (8)–(11).

(8)

In the very centre of the lake stood a mighty water wheel, known as the Wheel of the Cycle of Life.

(9)

A family trip is different from a bike trip with friends.

(10)

He does anything at all, he is held up as a monster and evil.

(11)

So we will see … Aussie comes with daily chart.

All sentences that were found to be irrelevant by at least one of the annotators were removed. In Figure 2, we present an instance of the ALVA annotation system used for the annotation of an utterance.

In the study reported here, the annotations were self-paced. When the annotator had classified the utterance, he/she ticked the submit box and a new utterance appeared in the box. It was not possible to go back to previous utterances to reconsider and revise the annotation. Two annotators, one with a Licentiate degree in English Linguistics and the other with a PhD in Computational Linguistics, carried out the annotations independently of one another. They followed the protocol that is schematically presented in Figure 3.

Figure 3:

The protocol followed in the annotation process of the BBC.

A manual with information about the annotation process, basic knowledge about the stance framework and the annotation tool was distributed to the two annotators. After studying the stance annotation manual carefully, the annotators participated in a seminar given by a senior linguist expert, where questions about the categorization and the process were addressed, and all outstanding issues resolved. The rationale of the annotation scheme was explained in detail on the basis of the information in Table 2 and the examples were discussed. Before the annotation session, there were five pilot rounds, each consisting of 100 utterances from our data set. After the pilot round, the annotators were given the opportunity to discuss their annotations with the senior linguist expert in order to ascertain that they had understood the task correctly and were able to work in accordance with the categorization scheme. The annotation procedure included two rounds of annotation by each of the annotators. The outcome of this procedure is evaluated and discussed in Section 4.2.

4.2 Annotation results

In order to evaluate the annotation results, we used two metrics: the inter-annotator agreement, which calculates the agreement between the annotators’ decision about an utterance, and the intra-annotator agreement, which tests the annotator’s second decision in relation to the first one about the same utterance. In the final version of the annotated corpus, the annotation disagreements between the two sets were discussed and problems resolved, and one of the two conflicting annotations was chosen, resulting in a gold standard version of the annotated corpus. In Table 3, the inter-annotator and the intra-annotator agreement sets are presented, as well as the gold standard corpus. All results reported in this section are based on the gold standard data.

Table 3:

The inter- and intra-annotator agreement sets.

Stance categories	Number of annotated utterances
	Inter-annotator agreement set (first/second annotator)	Intra-annotator agreement set (first/second decision)	Number of utterances in gold standard
agreement/disagreement	28/8	12/25	50
certainty	33/42	47/45	84
contrariety	109/127	151/214	352
hypotheticality	85/75	81/85	171
necessity	91/97	94/98	204
prediction	139/83	135/141	252
source of knowledge	100/114	148/113	287
tact/rudeness	16/27	14/14	44
uncertainty	121/78	89/86	196
volition	24/25	12/13	42

In Table 3, the output of the annotation process is presented. The first column shows the number of utterances annotated for each of the categories by each annotator. The second column presents the number of utterances attributed to the stance categories after the first and after the second rounds by the same annotator. Finally, the third column shows the final gold standard corpus after the annotation process, namely the number of annotations attributed to each stance category. In Figure 4, we illustrate the distribution of the stances in the BBC according to their annotation.

Figure 4:

The distribution of the stance categories in the BBC in descending order.

Figure 4 shows the distribution in terms of frequency of the different stances in the BBC. As discussed in Section 4.1, the utterances may express more than one stance type. In the gold standard corpus, 874 utterances out of 1,682 utterances in total were annotated with two stance categories. In 252 cases, three categories were attributed to one utterance. In 61 cases, utterances were annotated with four different stance categories, and 11 cases with five categories. In Figure 5, we show the category combinations of annotated utterances where more than one stance category was attributed.

Figure 5:

The different categories co-occurrences, after the annotation process.

Figure 5 shows the results for the cases where the combination occurred ten times or more. The numbers in the stance category text boxes refer to the total number of utterances annotated as such (the first one) and to the number of the additional stance annotations (the second one). For instance, in 244 cases out of the 352 utterances annotated as contrariety, more stances are detected and annotated. We observe that the contrariety category co-occurs frequently (58 utterances) with source of knowledge, and less often with volition (11 cases). The high number in some cases of these co-occurrences, as in uncertainty (214 categories co-assigned) shows that more than two categories were attributed to these utterances, and consequently this is an indication of the relation or interaction among the stance categories. The labelled arrows show how the stance annotations combine and in how many utterances this co-occurrence was attributed.

Our next step was the evaluation of the annotation results. The metrics were used in order to calculate the agreement level of the annotations of the different categories in the BBC. For these experiments we used two agreement measures, the F-score (van Rijsbergen 1979) and Cohen’s kappa (Artstein and Poesio 2008). As confidence interval (95 %) calculations for both measures, a bootstrap resampling based on percentiles using a 10,000-fold resampling was used (Myers and Hsueh 2001). The mean F-score and kappa were based on the mean result from the 10,000 resampled folds. In Table 4, the results of these measures are presented for both inter- and intra-annotator agreement sets.

Table 4:

The F-score and the kappa for the inter- and intra-annotator agreement sets.

Stance categories	Inter-annotator agreement set		Intra-annotator agreement
	Mean F-score	Mean kappa	Mean F-score	Mean kappa
AGREEMENT/ DISAGREEMENT	0.21	0.20	0.58	0.58
CERTAINTY	0.45	0.42	0.67	0.65
CONTRARIETY	0.78	0.76	0.76	0.71
HYPOTHETICALITY	0.78	0.76	0.79	0.77
NECESSITY	0.77	0.75	0.79	0.76
PREDICTION	0.57	0.52	0.78	0.75
SOURCE OF KNOWLEDGE	0.53	0.47	0.72	0.68
TACT/RUDENESS	0.55	0.54	0.78	0.77
UNCERTAINTY	0.62	0.58	0.81	0.79
VOLITION	0.44	0.43	0.71	0.70

In Table 4, we observe that for the inter-annotator agreement set, the highest evaluation score appears in the cases of contrariety and hypotheticality annotations. Their F-score is up to 0.78, which is a level of high agreement among the annotators for these two categories. The lowest score, on the other hand, is observed for the agreement/disagreement category (0.21). For the intra-annotator agreement set, the best score appears in the uncertainty category, achieving 0.81 as F-score. Except for the lowest score, again, for the agreement/disagreement category (0.58), all other categories have high agreement scores. These results are discussed and assessed in Section 5.

5 Discussion

In this study, we annotated stance in utterances extracted from blog posts. Based on the semantic information of the utterance, the annotators determined which stance/stances was/were expressed among ten different categories. In many cases, the speaker’s positioning was complex, expressing multiple stances in the same utterance. In other cases, no stance was expressed according to our stance categorization scheme. The final gold standard corpus shows that it is a possible task to determine speaker stance in texts from social media sources, in this case blog posts, for stance based on notional, cognitive-functional categories, and that the stance category framework was sufficiently clear and intuitive to be used as a practical protocol for annotations. No major divergences between the annotators’ decisions about the same utterances were observed. This gives credibility to our annotated data and enables us to use them for multiple purposes in future research.

In this section, we discuss and evaluate the annotation process and the results. Concerning the distribution of the annotated utterances of the BBC, we showed in Section 4 that there was a difference between the stance categories with regard to how often they occur. Contrariety is clearly the most often used stance category by the political blog text authors. This suggests that contrast is an appropriate rhetorical device for expressing opinions in a more balanced way as in (12).

In (12), the speaker has a clear opinion, expressing it using a contrastive sequence, starting with I don’t disagree, followed by a clause, starting with the adversative conjunction but, creating a balance, resulting in a less blunt statement.

The second most frequently occurring stance category is source of knowledge. We confirm in our data the common practice of people to refer to the opinions expressed by others, which is a typical pattern in conversations about political issues. In (13), we can see how referring to others is a means used to back up arguments with facts from an authoritative source.

Prediction is the third most frequently occurring stance category in these data, and it appears to be an important component in talking about future elections, EU and politics in general, as in (14).

Less frequent, but still occurring relatively frequently, are the three categories necessity, uncertainty and hypotheticality. Utterances belonging to the necessity category often take the form of recommendations or as a rhetorical device in which the reader is directly addressed. Expressions of uncertainty are used to convey doubt. Hypotheticality is mainly expressed in conditional sentences, and after a closer look at the annotation data, these forms/structures confirm their hypothetical meaning and function.

The remaining four categories appear less frequently in the data set. Certainty is only half as common as its opposite category uncertainty. agreement/disagreement and tact/rudeness are more likely to occur rarely due to the non-dialogic nature of the text type in this study. Expressions that indicate that these two categories are more likely to occur often when the text is directed toward a specific reader (in an online dialogue) and less often when speaker interaction is limited, as in the case of blog posts. This is, for instance, shown when we compare the occurrence frequencies obtained here to those in previous works (Skeppstedt et al. 2016), in which 31 % of the sentences extracted from debate forum posts contained expressions conveying agreement or disagreement. The final category, volition, might be infrequent in texts on political topics of this kind, and many of the utterances annotated as volition were wishes.

Moreover, it is important to note that due to utterance complexity more than one stance category were attributed to the 52 % of the BBC (874 utterances). In Figures 6–8, we show schematically the co-occurrences of stance categories in utterances. In Figure 6, we can see that agreement/disagreement frequently co-occurs with contrariety (in 12 cases out of 42 co-occurrences). In (15), we give an example showing that in utterances where the speaker agrees or disagrees he/she expresses a contradiction too.

In the right-hand chart of Figure 6, the co-occurrence of certainty with other stance categories is presented. We observe that certainty is primarily co-occurring with prediction (in 22 utterances out of 75 co-occurrences), with contrariety (14 cases) and with source of knowledge and necessity (10 cases). In the examples below, we show how these combinations may be worded and explain what made the annotators make their decisions.

Figure 6:

The stance categories that co-occur with the contrariety (on the left) and hypotheticality (on the right).

Figure 7:

The stance categories that co-occur with the necessity (on the left) and prediction categories (on the right).

Figure 8:

The stance categories that co-occur with agreement/disagreement (on the left) and with certainty (on the right).

In (16), the speaker expresses a prediction that Brexit is not going to happen in an assertive way, and the use of simply reinforces his or her position about the prediction he/she makes. In (17), the speaker juxtaposes two properties in a contrastive construction. The first part, including not just clear, triggers the readers’ expectations of a property that is stronger than ‘clear’ here expressed by overwhelming. This sliding along the scale, the contrastive adversative construction and the final extreme evaluation are indications of the speaker’s strongly assertive stance. In (18), the speaker alludes a reported source of knowledge reinforced by the assertive expressions the bottom line and did indeed. And in (19), in the first part of the utterance, certainty is expressed through it is clear, which takes scope over the rather assertive and imperative consequence expressed in the subclause. Certainty does not occur often with agreement/disagreement or hypotheticality. In the first case, we could assume that speakers, when expressing their agreement or disagreement do not use forms or structures of certainty, so they agree or disagree with something or someone in a gentler or milder way. Additionally, conditionals in hypotheticality express a low level of certainty by their semantic nature as being irrealis expressions.

In Figure 7, the stance categories that co-occur with contrariety and hypotheticality are presented. As we have already said, contrariety is the most frequent category, and it is also the category that co-occurs often with other stances. Its co-occurrence with source of knowledge is the most frequent one (in 58 utterances out of 244 co-occurrences), but also appears quite frequently with prediction and uncertainty (43 and 45 cases respectively), necessity (29 cases) and hypotheticality (26 cases), and less frequently with the other categories. Contrariety is the most interactive one with the other stances, and it is compatible with all of them. In (20)–(22), we show examples of the most frequent co-occurrences.

In (20), prediction is co-occurring with contrariety. The structure negation + but expresses contradiction and has also a predictive meaning. In the utterance in (21), we can identify source of knowledge, where the objective source statistics is employed to confirm the statement made by the speaker, which was annotated as contrariety. In (22), we observe the markers of uncertainty think and might in an utterance where contrariety and uncertainty co-occur.

Contrariety has a wide range of forms through which it is worded (and in addition to that it interacts significantly with other stances and their ways of being worded), and it appears to be a common construction for speaker positioning. The combinations of hypotheticality annotations co-occur mostly with contrariety (26 utterances) and with uncertainty (23 utterances). Conditional structures are often speculative and as in (24) include a comparative and contrastive, which enhance the role of these other two stances. Apart from the conditional structure that expresses hypotheticality, the lexical forms rather than and perhaps evoke contrariety (23) and uncertainty (24) stances.

In Figure 8, the combinations of necessity and prediction with other stance categories are presented. The necessity category is compatible with other stances, but most often it combines with contrariety, as in (25).

In the case of prediction, the most frequent co-occurrence is uncertainty (in 86 utterances) given that predictions are not facts or certain events, and cannot be adopted with confidence, see (26).

In (26), we find expressions of uncertainty (might, suggest) which attenuate the strength of the prediction and the related level of confidence.

As for the categories source of knowledge and volition, we see a similar distribution. Apart from a second stance that frequently co-occurs with the first category, not many utterances are annotated with co-occurring stances. Source of knowledge co-occurs most often with contrariety (58 utterances). In 11 cases out of the 42 utterances annotated as volition, the contrariety category was co-assigned. Not many co-occurrences are annotated with tact/rudeness. It is not possible to draw any conclusions about why this is the case because of sparsity of the data. Finally, the uncertainty utterances are mostly co-annotated with prediction, contrariety and source of knowledge. uncertainty is the most co-annotated category, where more than one stance is frequently attributed to the same utterance.

Concerning the annotation agreement results, we provide two potential explanations about the results. The inter-annotation results provide useful information about the efficacy and the reliability of the proposed framework. More specifically, a high degree of disagreement (low inter-annotator scores) between the annotators’ decisions in the annotation of one or more stance categories to the utterances may indicate either paucity of this stance in discourse, which may make its recognition a difficult task for the annotators (especially in the absence of explicit, straightforward cues such as if, so, but, etc.), or the inefficiency of the framework’s notional category for detection. The divergence among the annotator’s decisions could also be due to lack of linguistic items that could lead them to the detection of a stance in an utterance. Another clue that may explain the annotators’ disagreement can be the annotators’ different interpretations as a consequence of their intuitions about the semantic information that each utterance carries. In contrast, the agreement (high inter-annotator scores) between the two annotators in the attribution of stance is an indication of the reliability of the annotation scheme, or there were unambiguous linguistic clues that made the annotation decision easier and more evident for the annotators.

While the contrariety, hypotheticality, necessity and uncertainty categories showed good inter-annotation agreement score, the remaining six categories did not have high scores. In the intra-annotation agreement, we also observe differences among the different categories’ scores, but not to the same extent as the inter-annotation agreement results. The intra-annotation scores are on a similar level with the highest value found in the uncertainty category (0.81). All of them are over 0.65 with the only exception of the agreement/disagreement category, which is the lowest (0.58).

Regarding the F-score and the kappa measurements, the few occurrences of the utterances annotated for certainty, agreement/disagreement, tact/rudeness and volition led to very wide confidence intervals, and, therefore, no informative measures for intra- or inter-annotator agreement could be derived. Among the other six categories, for which more informative agreement measures were obtained, it can be concluded that for contrariety, hypotheticality and necessity, reasonable ^[5] agreement figures were obtained. For these three categories, there were no major differences across the results obtained for the inter- and the intra-annotator agreement. For uncertainty, prediction, and, in particular, source of knowledge, reasonable levels were obtained for intra-annotator agreement, but low inter-annotator agreement scores.

A critical comment about the corpus’ size could be that the BBC is a small text collection in comparison to automatically generated and annotated corpora. The extraction of blog posts was easily implemented using the Gavagai API, but the annotation of the sentences by human annotators was a laborious and time consuming task to perform. A great deal of previous work on stance annotation of text data has been about a binary decision, whether an utterance is for or against a specific topic. But, in the present study, the goal was to proceed in a way that better mirrors language understanding in natural communication focusing on a relatively free task carried out by language users on the basis of instructions describing cognitive-functional categories. The goal of this was to investigate whether it is at all possible to successfully make use of such categories instead of lexically driven tasks that only make use of form. We thereby took the initial steps to delve deeper into what stances speakers take and the way they are expressed in a sentence and thereby to highlight the dynamics of word meaning in discourse. It is clear that the annotation decisions depend not only on how the annotators perceived the semantic information in the utterances, but also on the speakers’ purpose and scope when they chose a specific form/structure for the creation of an utterance. Annotators are of course constrained by their own intuitions and general knowledge about how communication evolves on different occasions and in different contexts. They identified speaker stance in writing in each utterance and decided among ten different categories, which is a demanding task. For instance, the utterance in (27) should not be annotated with the certainty label as the speaker is trying to present someone else’s position about a topic.

In an automatic processing of (27), the annotation tool would select the certainty label mainly based on the word sure, while human annotators are able to distinguish such differences and understand the potential difference between the speaker’s actual positioning, which in this case is annotated as source of knowledge. The annotators had to take into account such cases and decide accordingly, and as a result of the procedure followed in our study, we can be confident that the output, the annotated corpus, is a carefully annotated data set that will be used for further research purposes and in various tasks.

6 Summary and conclusion

This interdisciplinary study is an experiment in annotating a corpus using cognitive-functional categories as the basis for speaker stance identification. We start by reviewing the linguistics and the computational linguistics literature in order to provide a comprehensive background of work on stance in both fields. This review of previous work is meant to promote synergies and to point to the plethora of terms used for stance in general. On the basis of this review, ten different notional stance categories were identified, defined and exemplified, namely agreement/disagreement, certainty, contrariety, hypotheticality, necessity, prediction, tact/rudeness, source of knowledge, uncertainty and volition. This information was then written up as a practical tool for corpus annotation (Table 2). In order to test the viability of annotating a corpus using cognitive-functional categories, we compiled a corpus of political blogs, the BBC, which is a novel linguistic gold standard resource. Using the ALVA system, two experts performed two rounds of annotation, attributing one or more stance labels to the utterances. We measured the annotation agreement scores in order to determine the reliability of the annotations, both inter- and intra-annotation agreements, and obtained good results, in particular for the intra-annotation agreement. The highest score for inter-annotation agreement was found for the contrariety and hypotheticality categories and for the intra-annotator agreement for the uncertainty category.

Through our methodology, we approach stance in a different way from many studies both in linguistics and computational linguistics, where researchers in most cases start with a set of words that they assume have a given meaning (at least most of the time), which is far from always the case as we have shown in this article. Words and structures through which people take stance are polyfunctional and capable of expressing more than one stance type. In this study, the point of departure for the annotators was notional in the sense that it was their task to identify which stance types were expressed by the speaker in each utterance. The BBC is a gold standard linguistic resource that can be useful for both theoretical and computational studies. Linguists can perform analyses of real language in use; computer scientists have a data set with predefined classes ready to perform stance classification experiments or other NLP and visual analytics tasks. The next step includes the development of automatic methods in stance identification based on how speakers take stance in utterances and what constructions in language they use to cue different stances.