Many research questions in the field of communication, linguistics and cognition are answered on the basis of manual analyses of data collections or corpora: collections of (transcribed) spoken, written and/or visual communicative messages. Although many different forms of corpus analysis are used (Krippendorff, 2019), the generic base can be defined as assigning interpretative categories to particular variables in the corpus. For example, particular words or expressions can be classified as having an intensifying meaning or as being ironic or metaphoric; gestures or pictures can be classified as representational or decorative; pitch patterns can be categorized as either expressing or lacking a feeling of knowing on the part of the producer; a particular interpretation of an artful poster or advertisement can be classified as ‘matching the intended meaning’ or not; a coherence relation between two utterances in a discourse can be labeled as semantic or pragmatic.

Obviously, these categorization tasks are extremely diverse in nature. For one thing, the span of the units can vary enormously: units can be sounds, words, phrases, utterances, images, complete texts or conversations. For another, the type of unit varies: annotating pitch in a telephone interaction is a completely different task from annotating the verbal language in that interaction, and the analysis of words is a task different from the analysis of gestures or images. It becomes even more complicated when the different modalities are combined. It is this variety in span and type of units that makes the analysis of discourse more intricate than the analysis that restricts itself to one linguistic level such as morphology and grammar. Despite the diversity, all of these activities fall under the heading of discourse coding: analyzing discourse in a systematic way in order to gain insight into underlying patterns. Often, this task is performed in a quantitative way, establishing or comparing frequencies of use, or trying to extrapolate frequencies from samples to populations.

In this kind of quantitative content analysis of discourse, the coding of subjective language data often leads to disagreement among raters. This is partly due to coding errors, and partly due to the inherent ambiguity of the language phenomena. Disagreement can occur even after an extensive training phase, even if an explicit code book is used that has been tested and adapted, even if the number of coding categories is limited, and even if experts are employed instead of naive and untrained coders (Spooren & Degand, 2010). Problems increase for the analysis of static and dynamic visuals in discourse. Often, several rounds of coding are necessary to reach a sufficiently high intercoder reliability statistic such as Cohen’s Kappa. At the same time, our publication outlets require that such a Kappa is reached after only one round of coding (cf. Neuendorf, 2002, p. 146). Allegedly, only with one round of coding, categorizations can be considered replicable and valid.

In most cases, the discourse units of interest come from a continuous stream of information. As a consequence, these units are often not predefined, and coders need to unitize the discourse before they can actually start categorizing these units. This may give rise to agreement issues as well, since coders can disagree about the number, length and alignment of the units in a discourse. For an analysis of the intricacies of unitizing issues and a valuable suggestion for quantitatively assessing the agreement on the unitizing effort, we refer the reader to Krippendorff (2019), Krippendorff et al. (2016) and Mathet et al. (2015). In the remainder of this paper, we focus on the issue of categorizing predefined units. Our restriction does not imply that we discard unitizing discourse, or the possibility that unitizing and categorization are interdependent, but we do believe that categorizing discourse units is a challenging task in itself, worth a separate discussion. Also worthy of a separate discussion is crowdsourcing. This can be seen as an interesting development that researchers make use of to have their data annotated by large numbers of naive coders, for example via ProLific (Suviranta & Hiippala, 2022). Another noteworthy omission is the possibility of biases in cross-cultural analyses (Peña, 2007; Quiros-Ramirez & Onisawa, 2015).

In our paper, we make use of the terminology in Table 1. For example, in her quantitative corpus analysis of metaphoric language use in news discourse, Pasma (2011) worked with a corpus of newspaper texts from two time periods (1950/1951, 2002; 80,000 words per period). The units of her analysis were the words in the newspaper texts. For each word, she assessed the coding category of the variable metaphoricity: a word was used metaphorically or not; in other words, the level of measurement was nominal. She assessed the intercoder reliability (ICR) by having a second coder perform the same discourse coding task and calculating an ICR statistic.

Our paper differs in several respects from related work like Bayerl and Paul (2011), who give a quantitative meta-analysis of factors influencing percentage agreement in a large number of linguistic corpus studies in different domains (word-sense disambiguation, prosodic transcriptions, and phonetic transcriptions). Our aim is to give a qualitative analysis of the factors involved, using case studies from various areas of discourse analysis. As a consequence, we are not limited – as Bayerl and Paul are – to the restricted number of factors available in the data set that is used.

In the next sections, we first sketch the scope of the reliability problem by describing different degrees of ‘wildness’ in discourse data. We then describe the tension between reliability and validity. Subsequently, we provide an overview of shortcomings of using Cohen’s Kappa scores as a measure of intercoder reliability – ICR from now on –, and suggest alternative statistical ICR metrics. Our paper continues with hands-on advice on how to improve ICR, and concludes with a reflection on where to go from here, with special attention to big data and multimodality.

The quality and outcome of corpus analyses, as well as the success of ICR tests, is determined by a large number of factors. In this section, we will discuss the effects of three important factors on the openness and ambiguity (‘wildness’) of discourse data: different ways of collecting data, different ways of establishing coding categories, and the kind of coding categories that are taken into account.

First, data collections can be elicited experimentally or selected from naturally occurring data. The latter tends to be more difficult to code reliably. For instance, Arts et al. (2011) asked their participants to produce referring expressions describing entities on a screen in a controlled setting. The result was an easy-to-code dataset of referential expressions containing only attributes visible on the screen. This differs sharply from the problems encountered while encoding the stylistic elements in naturally-occurring newspaper and web texts, such as the ones reported in Liebrecht (2015).

Second, coding categories can be established in different ways. On the one hand, they can have a predefined theoretical position and definition. Examples are using an enchiridion – a short handbook – to categorize intensifiers on a lexical base (Van Mulken & Schellens, 2012), or using an unambiguous and theory-based definition of verbal irony (Burgers et al., 2011). Low intercoder reliability could be prevented if researchers would study only variables with such well-defined categories that hardly cause any interpretation noise. On the other hand, coding categories can start from an intuition and emerge gradually and exploratively as the analysis proceeds (Van Enschot & Donné, 2013). Many interesting questions in the field of human communication are not ready for controlled types of research, and therefore require an explorative approach. Examples of such intriguing questions are: ‘What makes a visual message metaphorical (Forceville & Urios-Aparisi, 2009)?’ and ‘Which linguistic or audio-visual cues can we consider to be deceptive (Hancock et al., 2007)?’ The coding of the latter type of data tends to be much more difficult than the coding of the first type.

Note that agreement tests can be useful both in controlled and exploratory studies. Controlled studies require a high degree of precision and validity, and consequently only high levels of ICR outcomes are acceptable. In explorative studies, however, ICR scores can be used as a heuristic tool to objectify or specify individual intuitions, to try out coding category definitions or segmentation options in data collections. During this incremental process, lower ICR rates are acceptable, although one may want to eventually perform a more controlled analysis to validate the final version of the newly developed coding system.

Third, a set of coding categories can vary from closed to open. On the one extreme, the coding categories form a dyadic, mutually exclusive, closed set (for example yes-no, present-absent), clearly defined in terms of objective characteristics. On the other extreme, the number of coding categories is not fixed (for example ways to intensify an utterance) or the coding categories are not mutually exclusive (for example literal versus figurative language). Such coding categories leave room for an exploratory analysis, but at the same time pose problems for ICR.

Our discussion of these three factors illustrates that analytical data can be less or more wild, and that the degree to which our data are wild depends on the choices made by the researchers. The researchers make decisions about the type of questions asked (for example, do they focus on the question whether or not an utterance is subjective or whether a specific word indicating subjectiveness such as terrific occurs in the utterance), about the collection method used (for example, do they focus on a corpus of all tweets produced in a one-hour slot in The Netherlands, or on the one-word-production results of a word-listing task), about the theoretical framework and position taken (for example, do they start from the position that all subjective words can be divided into two or three major coding categories or not), and about the way in which theory is operationalized into coding categories (for example, do they define subjective words as a closed set of identifiable elements in discourse, or as an open-ended class).

These choices to a large extent determine the success of the coding task. Once we agree that data can be wild, the question is which precautions can be taken to make intercoder reliability tasks doable, not trivial, and successful or at least informative. We will address such precautions in the section on how to improve intercoder reliability.

Data quality is a matter of both reliability and validity. To illustrate the difference between reliability and validity, various scholars use the metaphor of a target or a dart board (Krippendorff, 2019; Trochim, 2006). Imagine your goal is to systematically hit the bullseye. Your performance is unreliable if the darts are randomly spread across the board. It is reliable if you consistently hit the same spot on the board – irrespective of whether this is the bullseye or some other spot, such as the ‘double 20’. However, your performance is only valid if you consistently hit the intended spot on the board, and not if you just hit the bullseye every now and then. Thus, the ‘dart board’ metaphor demonstrates that reliability and validity both need to be high, and that high reliability does not necessarily imply high validity. Similarly, aiming for high ICR scores is not a guarantee for good validity. On the contrary, it may even yield serious problems for the construct, external and internal validity of the research.

An established viewpoint is that reliability is at least a necessary condition for validity (Moss, 1994). What does this imply for researchers with ‘wild’ data that frequently result in insufficient intercoder reliability scores? One way out of this problem is to consider the theory or the coding procedure yielding the analysis of such unreliable data to be underdeveloped to such a degree that the researchers should go back to the drawing board. Alternatively, we could restrict the generalizability of our results to the limited set of data that we can code reliably. Such a solution is chosen by Liebrecht (2015) for the analysis of intensified language. She reports analyses on the subset of data on which both coders agreed. Of course, this limits the generalizability of the results. To accommodate that problem, she also reports findings for the intensifiers identified by each coder separately, and only draws firm conclusions when all results point in the same direction.

Below, we discuss the different types of validity, with special attention for how validity can be at odds with reliability. It will become clear that a high ICR – if achievable – is not always a guarantee for having valid data.

In this section, we turn to a selection of settings in which the most widespread intercoder reliability metric, Cohen’s Kappa, can be misleading: when each item is annotated by a different selection of annotators, when the categories are in an ordering relation, or when there is an imbalance in the frequency at which different categories are annotated. We provide an overview of metrics that can be applied alternatively to Cohen’s Kappa. We will focus on the main characteristics and advantages of these alternative metrics, without providing formulas.¹ The issues we describe all assume that the discourse units have already been established, that is: we describe the situation in which the unitizing stage has been finished, and the researcher has to categorize the units of interest.

Cohen’s Kappa was proposed by Jacob Cohen (1960) and takes into account the prior chance that two annotators agree on the annotation of any coding category. This makes Cohen’s Kappa a more realistic metric for intercoder reliability than percentage agreement, which can easily give misleading insights. For instance, one study using two coding categories has a fifty percent chance that coders agree, while another study using four coding categories yields a 25 percent agreement chance. As a result, given a similar percentage agreement, the first study will yield higher ICR scores than the second study simply due to chance (Artstein & Poesio, 2008, pp. 558-559).

As a way to account for chance agreement, Cohen’s Kappa presumes all items in a set to be coded by the same two annotators who are considered statistically independent. The chance for annotator 1 to choose certain categories is calculated, as is the chance for annotator 2 to choose certain categories. Based on these chances, the possibility for two annotators to agree coincidentally is calculated. As Krippendorff (2011) explains, this is not a realistic perspective on the role of chance in ICR: Annotators are dealing with the same categories to code and have read the same code book, and hence their annotations should not be seen as independent sets. A consequence of this is that the Cohen’s Kappa metric does not penalize patterns where the class distributions of the annotators are highly distinct. For example, if annotator 1 has chosen category A for 80 % of the items and annotator 2 has only chosen this category for 40 % of the items, the chance that both agree on this category is (0.8*0.4=) 0.36, and the chance that the two agree on the other category is (0.2*0.6=) 0.12, resulting in a chance agreement of (0.36 + 0.12=) 0.48. This is a smaller penalty than when both annotators would for example choose category A for 40 % of the times (a change agreement of 0.52). It would make more sense to look at the combined decisions of all annotators as a proxy of the class distribution, which is done in metrics like Fleiss’ Kappa (Fleiss, 1971), Scott’s pi (Scott, 1955) and Krippendorff’s Alpha (Hayes & Krippendorff, 2007; Krippendorff, 2019). These metrics consider the proportion of times that each coding category is chosen by annotators, as well as the agreement per single item. Essentially, annotators are then seen as interchangeable, which is more in line with the notion of reliability of annotations: if the annotations in the end rely on a straightforward decision procedure which is reflected in the agreement, it should not matter who does the annotations and any one annotator could be replaced by another without much effect on the agreement.

An important property of the coding task is the measurement level of the variable. In standard form, the Cohen’s Kappa metric presumes a nominal scale, in which no ordering exists between the coding categories. It will give wrong insights when applied to other than nominal variables, with ordinal, interval or ratio levels. With these levels of measurement, it should be penalized when two coders annotate coding categories that have a larger distance to one another. Metrics that do apply such a penalization are the Weighted Cohen’s Kappa (Cohen, 1968) and Krippendorff’s Alpha (Hayes & Krippendorff, 2007; Krippendorff, 2019). In these metrics, the agreement, or disagreement in the case of Krippendorff’s Alpha, for any pair of levels is weighted by taking into account the distance between the categories, such that more distanced categories add a lot less to the agreement score, or a lot more to the disagreement score. Krippendorff’s Alpha also allows for missing data points, by considering the total number of annotations that were made.

Another factor that needs to be considered when assessing intercoder reliability is data skew: the degree to which data is asymmetrically distributed over coding categories. Di Eugenio and Glass (2004) and Jeni et al. (2013) show that Cohen’s Kappa and Krippendorff’s Alpha are highly sensitive to imbalanced variables: the agreement score will drastically decrease with a bigger data skew. The reason is that the prior chance of annotators making similar annotations is high if there are one or a few dominant coding categories. As the higher chance agreement is subtracted from the percentage agreement, a high percent agreement may still result in a relatively low ICR. In other words, it is unlikely to reach a high Cohen’s Kappa or Krippendorff’s Alpha when most of the items fall under one coding category. A solution is to calculate the Kappa Max (Umesh et al., 1989), which returns a Kappa value that is relative to the upper bound of the Kappa that follows the strict chance agreement. Alternatively, adaptations have been suggested to correct Kappa for such biases, such as PABAK (prevalence-adjusted bias-adjusted kappa; Byrt et al., 1993).

The Mutual F-score is another metric that can be used to mitigate the influence of category imbalance. Mutual F-scores are based on F-scores, which are often used in Information Retrieval and Machine Learning for system evaluation (Van Rijsbergen, 1979). In contrast to the other metrics, the Mutual F-score estimates the agreement per coding category rather than over the total of annotations. Considering the annotations of one coder as ground truth, for each category the annotations of a second coder can be assessed with respect to this ground truth. At the basis of the F-score are three counts: of True Positives (TP), False Positives (FP) and False Negatives (FN). TP s are instances that both coders annotate with the coding category in focus, FP s are instances that are only annotated by the second coder with the coding category in focus and FN s are instances that are only annotated by the first coder with the coding category in focus. Based on these values, Precision can be calculated as the percentage of instances coded by the second annotator with the coding category in focus, that were actually correct with respect to the ground truth annotations of the first coder (TP s divided by the total of TP s and FP s). Recall can be calculated as all instances that were annotated by the first annotator with the coding category in focus, that were annotated in the same way by the second annotator (TP s divided by the total of TP s and FN s). The F1 score is the harmonic mean of Precision and Recall: 2*(Precision*Recall)/(Precision+Recall). To reach the Mutual F-score between annotators, the role of the ground truth annotator is swapped, and the F-score is computed again in the other direction. Then, Mutual F-score can be computed as the mean of the two directional F-scores. As the annotations of other categories by both coders are left out of this calculation (the TN s, true negatives), the Mutual F-score allows to zoom in on the agreement per coding category.

To illustrate the informativeness of the Mutual F-score in comparison to other metrics, we provide an example from our own work. Kunneman et al. (2015) aimed to automatically detect sarcasm in Twitter posts, and trained a model based on tweets that are accompanied by Dutch hashtags likely deployed by the sender to signify the sarcasm in their post. To validate this approach, for a random sample of 250 messages that include such a hashtag, three annotators coded whether they actually were sarcastic (Kunneman et al., 2015, p. 505). The confusion matrix for one of the coder pairs is displayed in Table 2.

The percentage agreement between the two coders is 91 % ((205 + 22)/250), but due to a dominance of the coding category Sarcastic, both Cohen’s Kappa and Krippendorff’s Alpha are 0.60. Considering the annotations of coder A as ground truth, the F-score for the coding category Sarcastic is 0.95 (TP = 205, FP = 11, FN = 12, Precision = 205/(205 + 11) = 0.95, Recall = 205/(205 + 12) = 0.95, F1 = 2*((0.95*0.95)/(0.95 + 0.95)) = 0.95) and the F-score for the coding category Non-sarcastic is 0.66 (TP = 22, FP = 12, FN = 11, Precision = 22/(12 + 22) = 0.65, Recall = 22/(11 + 22) = 0.67, F1 = 2*((0.65*0.67)/(0.65 + 0.67)) = 0.66). Given that the dataset of this variable contains two coding categories, the Mutual F-score on the coding category Sarcastic is therefore 0.95, and for Non-sarcastic it is 0.66. These two values highlight the difference in agreement on the two categories, where the less frequent coding category is the more problematic.

After having arrived at a reliability metric, the next step is interpreting this metric. Earlier on, we suggested that the ICR depends on the nature of the data and the research objective(s). This suggestion would imply that an ICR metric should be interpreted relative to these objectives. Interpretation may vary per type of study: for hypothesis testing, a higher ICR is required than for explorative studies. Similar suggestions can be found in Grove et al. (1981) and Spooren and Degand (2010) (see also the discussion in Artstein & Poesio, 2008, and Mathet et al., 2012).

In light of these examples, it is important to fully understand the mechanisms of a metric when interpreting its outcomes and be aware of how Cohen’s Kappa can mislead. Next to the overview in this section, Artstein and Poesio (2008) give an excellent, elaborate overview of the mathematics and assumptions behind the ICR metrics, and discuss issues of annotator bias (not all coders use each category equally often) and prevalence bias (some categories are used more often than others). Such overviews enable researchers to interpret the ICR metrics correctly, including Cohen’s kappa, allowing them “to make an informed choice regarding the coefficients they use for measuring agreement” (Artstein & Poesio, 2008, p. 590). The discussion also shows that existing heuristics for interpreting outcomes that do not take into account the context of annotation, seem too simplistic. In line with Perreault and Leigh (1989) who state that “different indices reflect different aspects of reliability” (p. 146), we recommend assessing interrater agreement with several metrics. This way, we can achieve a more complete interpretation of the factors in play.

Researchers need hands-on advice and practical solutions to ICR problems. In this section, we meet this need and suggest a number of concrete ways to improve intercoder reliability: making categories independent (5.1), reducing the number of categories (5.2), decomposing the analytical process into smaller steps (5.3) and some procedural measures such as optimizing the coding instructions (5.4). Some of these topics are also discussed by Bayerl and Paul (2011), who used a quantitative meta-analysis to identify several factors influencing reported agreement values (like number of annotators and number of categories in a coding scheme). Our discussion is more qualitative, allowing us to discuss factors not touched upon by Bayerl and Paul such as the decomposition of the coding process and the independence of the categories. Bayerl and Paul’s discussion is limited to three relatively local domains (word-sense disambiguation, prosodic transcription, phonetic transcription), whereas our discussion focuses on the level of discourse data. Other useful sources are for instance Krippendorff (2019), Neuendorf (2002) and Selvi (2020).

For scholars of discourse studies using quantitative content analysis, issues of intercoder reliability are of the highest importance, both for practical reasons (how do we convince our peers that our studies are worthwhile despite our wild data yielding low ICR scores?) and for methodological reasons. In this paper, we have discussed how discourse data can differ in their degree of wildness and – with that – in their risk of a low ICR, how reliability can be at odds with validity, and what to pay attention to when assessing intercoder reliability statistically. We have also demonstrated that intercoder reliability issues are not insurmountable. Implementing the measures as discussed in the previous section will help improve the ICR scores of categorizing predefined units. Since assigning categories to discourse units is an essential part of the discourse coding process, we also believe that these measures will be beneficial to the coding tasks that combine unitizing a continuum and categorizing these units (for example Mathet et al., 2015). Nevertheless, we see important challenges for future work, amongst which big data and multimodal discourse.