Token-based typology and word order entropy: A study based on Universal Dependencies

2.1 Dependencies and co-dependencies

The case studies presented in this paper are based on the frequencies of different word order patterns in corpora annotated using the Universal Dependencies (UD) approach. The cross-linguistic syntactic categories and parts of speech used in the UD framework are the result of a long evolution. One of the main goals and challenges of the UD project is to keep the annotation design satisfactory both for language-specific analyses and for typological comparison. ^[2]

I use the frequencies of so-called heads and dependent elements, as well as some co-dependencies (e.g. subject and object of the same verbal predicate). The full list is given in Table 1. This selection includes the main dependencies in nominal phrases, verbal phrases, simple clauses and complex sentences. Nominal and pronominal subjects, objects, obliques and modifiers of nouns were treated separately, since the order of nominal and pronominal constituents is often different. Subordinators in adverbial and complement clauses were extracted separately, as well, and so were adverbial modifiers of adjectives and verbs. Note that in the case of interclausal dependencies, the head is the predicate of the main clause or the head noun (for relative clauses), and the dependent element is the predicate of the subordinate clause. The dependencies related to punctuation, coordination, various discourse markers, vocatives and multiword expressions were not taken into account. I also excluded the so-called root (the predicate of the main clause), which does not depend on anything, and dependencies which are infrequent in the UD sample of languages (e.g. classifiers). Some contextual restrictions were made, as well. In particular, questions and exclamatory sentences were disregarded. Subjects, objects and obliques were only counted when they occurred in the main clause.

Note that the understanding of heads and dependents in the UD corpora is different from some syntactic theories in that functional elements (adpositions, auxiliaries, subordinators) are coded as dependents, not as heads. ^[3] One may agree or disagree with that approach, but it plays no role in the analyses that follow. The reason is that the central measure in this study, namely, entropy of the order of two elements X and Y, remains the same regardless of whether X is the head and Y is the dependent, or the other way round (see Section 2.2 for more details).

2.2 Shannon entropy

The main measure used in this study is Shannon entropy (Shannon 1948). It represents variation of word order in the twenty-four dependencies and co-dependencies described in the previous section. For each of the word order patterns in a corpus, I computed the entropy using the formula in (1):

where X is a binary variable representing two possible word orders, e.g. Determiner + Noun and Noun + Determiner. P (x_i) is the probability of one of the orders, which equals its relative frequency (proportion) in a given corpus. If the proportion of one word order (e.g. Determiner + Noun) is 1, and the proportion of the reverse order (e.g. Noun + Determiner) is 0, or the other way round, the entropy H is equal to zero. There is no variation. If the proportion of each of the possible word orders is 0.5, the entropy takes the maximum value of 1. If both orders are attested, and one of them is more frequent than the other, then the entropy lies between 0 and 1. For example, if the proportion of Determiner + Noun in a specific corpus is 0.9 (or 90%), and the proportion of Noun + Determiner is 0.1 (or 10%), then the entropy of this dependency is H = – (0.9 × log₂0.9 + 0.1 × log₂0.1) = 0.47. These numbers suggest that the relationship between probability and entropy is non-linear. As shown in Figure 1, this is indeed the case. Already a small amount of variation is sufficient to obtain relatively high values of entropy.

Figure 1:

Non-linear relationship between probability of a word order pattern A followed by B (as opposed to B followed by A) and entropy.

Entropy can be computed at different levels of granularity. For example, one can use the level of abstract dependencies, as shown above, or compute entropy for individual wordforms that represent dependencies, e.g. the entropy of actively_ADV + VERB and VERB + actively_ADV, where actively is an adverbial modifier of a verb. This approach will be discussed in Section 4.

Word order entropy, as it is defined here, is a purely descriptive and data-driven measure based on actual word order patterns. Its most important advantage is that it is objectively defined, unlike such notions as word order “freedom” or “flexibility”, which usually reflect intuitions about the acceptability of different word order variants in a pre-defined set of contexts. Although such intuitions may be of substantial interest, this subjective kind of variability is much more difficult to define, measure and compare cross-linguistically than corpus-based entropy.

It is also necessary to mention a methodological caveat. One of the main challenges of working with cross-linguistic corpora is their representativeness and comparability of the text types (cf. Croft 2003: 112). There are no multilingual corpora at the moment that could be compared to the carefully sampled British National Corpus or similar standards. The existing multilingual corpora can represent one text (e.g. Le Petit Prince used by Stolz et al. 2017) or numerous texts of a few rather specific genres (e.g. OPUS by Tiedemann 2012 with film subtitles, European Parliament transcripts and technical documentation). Comparable spoken corpora are also available, but for a limited number of languages (e.g. Haig & Schnell 2016a; Dingemanse et al. 2013).

To what extent text types are important for cross-linguistic investigations of word order entropy is an empirical question. In order to check if entropy scores are strongly influenced by text types, I compared different text types in eight languages from the Leipzig Corpora Collection (Goldhahn et al. 2012), ^[4] which consists of comparable corpora in diverse languages. The procedure and the results are described in the Appendix. The comparison suggests that the differences between different text types in one language are small.

3.1 Data for Case Studies 1a and 1b

This section discusses word order entropy in the Universal Dependencies corpora 2.1 (Nivre et al. 2017). This is a collection of 102 treebanks (i.e. syntactically parsed corpora) representing 60 languages from 24 genera and twelve different families, including one sign language and several isolates (Basque, Japanese and Korean). More details can be found in the Supplementary Materials: UDCorpora. First, the frequencies of every (co-)dependency in each possible order (e.g. Determiner + Noun and Noun + Determiner) in every corpus were extracted with the help of a Python script. If a language was represented by several corpora, all counts were summarized. Dependencies with frequency less than 20 (usually due to the small size of some corpora) were excluded from the subsequent analyses. Next, the proportions of word order pairs (e.g. Determiner + Noun and Noun + Determiner) were computed for each language and (co-)dependency. Finally, the entropy scores were computed as described in Section 2.2.

3.2 Case Study 1a. A probabilistic typology of languages based on word order entropy

This subsection presents a language classification based on word order entropy. Several languages with small frequencies were excluded (Buryat, Cantonese, Korean, Kurmanji, Sanskrit and Swedish Sign Language). The entropy values of the 24 (co-)dependencies were averaged in every language. Figure 2 presents the mean entropy scores for the languages in the UD corpora. The horizontal axis represents the average entropy of the head-dependent patterns. The vertical axis shows the average entropy in the order of co-dependents (SO/OS and OX/XO, where X stands for oblique). The higher the score, the more variable the word order is on average in a given language. Trying out combinations of other parameters (e.g. entropy of function words or clauses) did not reveal additional interesting patterns or clusters. The reason is that most of the entropy values are positively correlated. ^[5]

Figure 2:

Mean head-dependent order entropy and co-dependent entropy in the UD corpora.

Figure 2 shows that quite a few languages have high entropy on both dimensions (the top right corner). These are morphologically rich European languages (Basque, Finnic and Slavic), including several ancient Indo-European languages (Ancient Greek, Gothic, Latin and Old Church Slavonic). They are followed by the more analytical Romance, Germanic and Semitic languages, which have moderate scores. Notably, if languages have very low entropy, they have either low variation of the head-dependent patterns, or low variation of the co-dependent patterns, but not both. In particular, the Altaic and Dravidian languages and Japanese have very low entropy of head-dependent patterns, but the order of co-dependents is to some extent variable (cf. Jamashita & Chang 2000). In contrast, Irish, Coptic and especially Mandarin have the most rigid orders of co-dependents, but moderately rigid orders of heads and dependents, in comparison with the other languages. One may wonder if there exists a language in the world that allows no variation in word order at all.

How can we interpret this variation? Let us begin with the horizontal dimension, which shows the head-dependent entropy. Many morphologically rich, highly synthetic languages are located on the right. The analytic languages tend to be located close to the middle. The left-hand part of the figure with very low entropy includes verb-final languages (Turkish, Japanese, etc.), which are synthetic, as well (e.g. Hawkins 2014: 139–146). This supports the results presented by Futrell et al. (2015), who found that languages which have highly predictable head direction (i.e. low head – dependent entropy) tend to be mostly head-final. The low variation of the head-dependent orders in verb-final languages may be a manifestation of their ‘tight fit’ with regard to argument and predicate frame differentiation (Hawkins 2014: Section 7.2). OV languages usually have a morphological case system, narrow semantic range of basic grammatical relationships, no raising or wh-extractions. All these restrictions, together with rigid word order, help the hearer to identify the argument structure early, so that there is no need to make corrections after the verb comes.

The vertical dimension reflects mostly variation in the order of subject and object (SO entropy) because the entropy of objects and obliques (OX) is higher than 0.6 in most languages (with the exception of Mandarin, Coptic and Irish). The languages closer to the top have rich nominal morphology. For example, Lithuanian, an extremely conservative Indo-European language as far as the nominal declension system is concerned (Erhart 1987: 129), has the highest score on the vertical dimension. This suggests that high SO entropy is associated with the presence of case marking.

Indeed, there is a common view that rigid word order can ‘step in’ when there are no morphological cues that help to disambiguate between the main arguments, and that the availability of case morphology is associated with more flexible word order. This relationship is well-known in linguistics (e.g. Sapir 1921: 66; Jakobson 1971[1936]: 28). For example, Blake (2001: 15) considers word order ‘an alternative to case marking’ in languages like English, Indonesian, Thai and Vietnamese. Those languages in which the main arguments (A and O) are not distinguishable by means of case flagging or indexing will tend to have on average more rigid word order of the arguments than the languages in which A and O are distinguishable. Typological data in support of this idea have been provided by Siewierska (1998) and Sinnemäki (2008). Corpus evidence can be found in Futrell et al. (2015), who show that languages with flexible word order tend to have case marking. One can also find experimental evidence in Fedzechkina et al. (2016).

The corpus data allow us to formulate and test a more nuanced hypothesis than the previous accounts. The hypothesis can be formulated as follows: the amount of entropy in the order of S and O negatively correlates with the amount of formal overlap between S and O wordforms. In other words, the more confusable the forms on average, the more rigid the word order. It is more specific than the previous ideas because existence of case marking in a language does not guarantee that the forms are always distinctive. Many languages exhibit case syncretism in specific nominal forms or differential marking of subject and object, which depends on the semantic and pragmatic properties of the arguments (see the quantitative data in Levshina 2018: Ch. 6). Therefore, the categorical variable [+case] or [-case] may not be exact enough to represent the actual need for disambiguation.

To test the hypothesis, I computed for each language the number of individual nouns that were identical in the subject and object functions. I also compared the nouns in the function of an object and an oblique nominal phrase. Case marking with adpositions was taken into account on a par with case inflections, so that the Spanish form a Juan was just as different from Juan as the Russian accusative form Ivan-a is different from the nominative form Ivan. Only non-plural forms were taken into account, i.e. those that were not marked as ‘Plural’ in the morphological properties slot. Two confusability indices were computed. The first one was the proportion of lemmata that have identical forms as subjects and objects in the total number of lemmata found both in the subject and object positions in a given language. The second one was the proportion of lemmata with the same forms in the object and oblique functions.

The analyses revealed very little overlap between object forms and oblique forms. The languages differ mostly in the degree to which they distinguish formally the subject from the object. This variation can be seen in Figure 3, where the proportion of formally identical subjects and objects (per lemma) is plotted against the entropy of subject and object in a given language. ^[6] Overall, we see a negative correlation between the average confusability of the subject and object and the entropy of their order. A Generalized Additive Model with the genera as random intercepts reveals a non-linear negative effect displayed as the curved line in Figure 3 (p = 0.0003 for 2.8 estimated degrees of freedom). ^[7] The shaded area is the 95% confidence band. The model explains the data well (83.1% of deviance, i.e. variation, is explained; the adjusted R², which penalizes a model for having too many parameters, is 0.74). The model output can be accessed in the Supplementary Materials: Regression Output. Importantly, the correlation is between continuous variables, rather than binary features, such as [± case] or [± fixed word order]. This kind of relationship can only be tested on the basis of corpora.

Figure 3:

Negative correlation between SO entropy and the proportion of identical S and O forms.

Interestingly, the top right corner in Figure 3 contains no languages. This is the area with high confusability of Subject and Object, and with high variability of their order. ^[8] Compare that with the bottom left corner. There are no languages with zero confusability and zero entropy, but a few languages have low values on both dimensions (i.e. Japanese, North Sami, Persian, Arabic and Coptic). This suggests that the relationship is not only a correlation, but also to some extent a one-way implication: high confusability implies low entropy, but low confusability does not necessarily lead to high entropy. Similar ideas have been expressed on the level of languages as types (e.g. Kiparsky 1997; McFadden 2003). ^[9] This finding can be explained by diachronic processes: loss of case marking leads to word order fixation due to avoidance of ambiguity, as happened in Middle English (e.g. McFadden 2003), but development of case marking does not (necessarily) lead to more flexible word order. As Kiparsky put it (1997: 490),

A language may lose its inflections, but it cannot ‘lose its word order’ in the same sense: it must go on putting one word after another, even when it does not grammatically exploit or constrain word order.

Obviously, more research is needed in order to gain a full picture, which should also include the role of agreement markers, intonation and other resources for disambiguation.

3.3 Case Study 1b. Entropy of individual word order patterns

This section discusses variation of the word order patterns within and across languages. Two types of average measures were computed. The average intra-linguistic variability of an individual (co-)dependency is represented by its mean entropy. In order to take into account the hierarchical structure of the UD corpora, where some genera are represented by many languages and some by only one, I averaged the entropy first within each genus, and then computed mean entropy scores across the genera.

The cross-linguistic variability of a specific word order pattern is represented by the standard deviations of proportions of the word orders across the UD genera. They are based on the average proportions of the dependent element being before the head (e.g. the average proportion of auxiliaries before the main verb in a genus), and of the SO and OX order. These proportions were first computed for each language, and then averaged for each genus. Finally, the standard deviations were computed that reflect the cross-linguistic variability of the proportions across the genera.

These two types of variability, intra-linguistic and cross-linguistic, are shown in Figure 4. The horizontal axis displays intra-linguistic variation of word order in the (co)-dependencies, measured as mean entropy. The patterns on the left-hand side have low entropy. This means that the order is fixed in most languages. For example, a language has either prepositions or postpositions. The right-hand side of the plot contains the word orders with greater intra-linguistic variability. In an individual language, the position of pronominal objects and obliques, adverbial clauses and adverbial modifiers of verbs, obliques with regard to objects and predicates tends to be the most flexible, whereas the position of complementizers, adverbial subordinators and adpositions is rigid in most languages. Various modifiers of nouns (adjectival and nominal modifiers, determiners, and attributive clauses) usually have limited variability, as well as auxiliaries, copulas, pronominal subjects, complement and subject clauses. Nominal subjects and numeral modifiers are more variable.

Figure 4:

Intra-linguistic variability (horizontal axis) and cross-linguistic variability (vertical axis) of selected syntactic dependencies from the UD corpora.

The vertical axis represents the cross-linguistic variability of the word order patterns specified by the labels. The numbers show the standard deviations of the proportions of the corresponding word orders. Adpositions, adjectival modifiers, subordinators, auxiliaries, objects, as well as attributive, complement and subject clauses, display substantial cross-linguistic variation, as one can judge from their position at the top of the plot. In contrast, the languages agree with regard to the position of subjects, determiners, numerals and adverbial modifiers, which are located below.

This information has important implications for word order typology as a domain of inquiry. The patterns in the top left corner of the plot include the ones that have received the greatest attention in the literature: adpositions, nominal objects, adjectives and nominal modifiers (predominantly genitives). This figure explains why: these patterns exhibit high cross-linguistic variation, but low intra-linguistic variation, which makes them perfect candidates for typological investigations based on reference grammars. Although this bias is understandable from a practical point of view, it also means a considerable data reduction when one focuses only on those patterns (cf. Wälchli 2009).

The patterns in the bottom, on the left, may be less interesting for language classification because they exhibit less variation, but they are still important for typological purposes because they allow us to formulate cross-linguistic generalizations. Most languages tend to put subjects before objects. This is not new (e.g. Hawkins 2014: 5). What is more interesting, perhaps, is the fact that the variation in the position of pronominal subjects with regard to verbs within a language is lower than the variation in the position of nominal subjects. This may have to do with the fact that nominal subjects can be used to introduce new information, as in the example In the middle of the room stood a table. In contrast, pronominal subjects are topical and given, and are used before the verb in topic-first languages (cf. Givón 1984: Section 6.5; see also Section 5.1). Also, the order of numerals, determiners and adverbial modifiers of adjectives is to some extent similar cross-linguistically. They tend to occur predominantly before their heads. However, a closer look reveals that this is an artefact of the UD language sample, as far as the order of numerals and determiners is concerned. According to Dryer (2013a, 2013b), numerals and demonstratives (a subclass of determiners) exhibit strong areal patterns. They are pre-nominal in Eurasia (where most of the corpora come from), but can be post-nominal in some other areas, such as Africa and Southeast Asia. The low intra-linguistic entropy of determiners, however, is quite informative. It can be explained by their high level of grammaticalization and loss of syntagmatic variability (cf. Lehmann 2015: Section 4.3.3; see also Section 5.1).

Finally, the obliques and adverbials are located on the right, which means that they exhibit high entropy in individual languages. This variability can be explained by their different functions in discourse. For example, adverbial clauses of condition tend to precede the main clause, while adverbial clauses of result and purpose usually follow the main clause (Diessel 2001). This is explained by universal cognitive and discourse-pragmatic factors, in particular, by iconicity of temporal order. Similar explanations can also be offered for obliques and adverbial modifiers of verbs. For instance, a closer look at English, Finnish and Russian obliques and adverbials reveals striking similarities between the languages. The adjuncts on the left from the predicate are often those which introduce causal links with the previous discourse (e.g. therefore, thus), mark the relative position of the statement in the rhetorical structure (e.g. also, moreover, finally), express the epistemic stance towards the entire preposition (e.g. possibly, reportedly, certainly) or the speaker’s emotional attitude (e.g. hopefully). The adjuncts on the right often have the directional meaning (go + abroad, to the market), expressing the potential outcome of the action. These word order patterns have received less attention in typology than the others, but they are also important because they allow us to formulate probabilistic universals based on the general discourse-pragmatic principles.

To summarize, although word order typology has mainly focused on the patterns with low intra-linguistic and high cross-linguistic variability, we also need to explore the other patterns. It is not unreasonable to expect that the high-entropy patterns can be particularly valuable for providing a window into the universal cognitive and communicative biases.

4.1 Aims of this section

In the previous section it was mentioned that high entropy of particular word order patterns may be explained by their diverse functions in discourse. This functional diversity can be approximated by taking into account the individual wordforms. For example, on Monday is typically a temporal adjunct, whereas therefore is a causal adverbial modifier. To what extent will the variability of the word order patterns change if we compute the entropy on the lexical level? Will we see a completely different distribution of languages and individual dependencies? The goal of this section is to answer these questions.

4.2 Data

I took large news corpora of eleven languages (100,000 sentences in each) from the Leipzig Corpora Collection (Goldhahn et al. 2012) and parsed them with the UDPipe software (Straka & Straková 2017), implemented in the R package udpipe (Wijffels 2018), which provides tokenization, lemmatization, part-of-speech annotation and syntactic parsing. The languages were Arabic, Basque, English, Finnish, Hindi, Indonesian, Irish, Mandarin, Russian, Tamil and Turkish. All of them belong to different genera. The reason for using the additional larger corpora is that one needs high frequencies in order to obtain estimates of entropy for individual lexemes.

From each of the corpora, I extracted the head-dependent patterns, using the same approach as outlined in Section 2, plus the information about the wordform of the dependent element, e.g. big + Noun, where big is an adjectival modifier amod, which modifies a noun. These types of patterns will be called here lexically specific dependencies. Only the dependencies in the simple clause were considered. The dependencies related to complex clauses were excluded because it is difficult to imagine that the lexical predicate of the subordinate clause can be highly relevant for the semantic and pragmatic function of the clause.

The wordforms of the dependent elements also included adpositions with the function ‘case’ because these functional units play an important role in determining the function of the lexeme in a sentence. For example, John, of John, John’s, to John, etc. were all considered different wordforms and were treated separately, similar to Russian Ivan, Ivan-a ‘I.-acc/gen’, Ivan-u ‘Ivan-dat’, k Ivan-u ‘to Ivan-dat’, etc. For each lexically specific dependency I computed the frequencies of occurrence of this wordform before and after the head (e.g. the form table as a nominal subject before and after the predicate). Only the wordforms with a frequency above 20 were considered. On the basis of those frequencies, I computed the measure of entropy for each individual lexically specific dependency using the method described in Section 2.2. In addition, the entropy scores for each syntactic dependency without the lexical information were computed, as was done in the case studies discussed in Section 3.

4.3 Results

Figure 5 displays the mean entropy measures for the eleven languages, based on the sixteen dependencies in simple clauses. The horizontal axis shows the mean entropies at the level of abstract dependencies without lexical information. The vertical axis displays the entropies based on the lexically specific dependencies. The entropy scores were first computed for each wordform, then averaged for each dependency (e.g. det_Noun, amod_Noun, etc.) and finally averaged across the languages.

Figure 5:

Mean head-dependent entropy in a language. Horizontal axis: mean head-dependent entropy without lexical information; vertical axis: mean head-dependent entropy with lexical information.

Importantly, the entropies based on the abstract dependencies are higher than the ones based on the lexically specific ones, as the reader can infer from the values on the axes. This means that some part of the variation can be explained by the variation between individual wordforms. At the same time, there is a strong correlation between the two sets of scores: r = 0.978133, p < 0.0001. This means that the entropy-based classification of the languages remains essentially the same, regardless of the level of granularity.

The languages in which the averaged dependencies show the largest differences are Finnish (the difference between the mean abstract dependency entropy and mean lexically informed entropy is 0.24) and Russian (0.23). The smallest differences are observed, as one would expect, in the low-entropy languages Hindi and Tamil (0.04 both).

As for the individual dependencies, consider Figure 6. Again, the horizontal axis shows the mean entropies at the level of abstract dependencies, whereas the vertical axis displays the mean entropies at the level of wordforms, averaged across the languages.

Figure 6:

Mean entropies of dependencies averaged across eleven languages. Horizontal axis: mean head-dependent entropy without lexical information; vertical axis: mean head-dependent entropy with lexical information.

Again, we find that the entropies are strongly correlated: r = 0.91, p < 0.0001. Adpositions, auxiliaries, adjectives and nominal modifiers of other nouns have low entropies regardless of the method. The highest entropy scores on both axes are observed for obliques and for adverbial modifiers of verbs, similar to the results reported in Section 3.3. However, the correspondence is not perfect. On average, the dependencies that show the greatest differences between the entropies with and without lexical information are advmod_Verb (the difference is 0.24), det_Noun (0.23) and nummod_Noun (0.21). These are the dependencies where the greatest functional specialization is expected. This is not surprising. Some examples of adverbial modifiers of verbs were given in the previous section. Their position often depends on their function. As for determiners, they constitute a very heterogeneous category: demonstrative, possessive, negative and indefinite pronouns, articles, etc. Their positions with regard to the head noun can vary. For example, in Irish, the definite article and possessive pronouns precede the noun, e.g. an ‘the’ + fear ‘man’ and do ‘your’ + chara ‘friend’, whereas demonstrative pronouns follow it, e.g. an ‘the’ + bhean ‘woman’ + seo ‘this’. As for numeral modifiers, they can also perform diverse functions. For example, they can specify the quantity, e.g. 5 books, but appear in dates (e.g. the year 2019), telephone numbers (e.g. the emergency number 112) or addresses (e.g. 10 Downing Street). The annotation of these instances in a corpus is not trivial. The dependencies with the smallest differences are aux_Verb and nsubjPRON_Pred (both 0.04). They also display low entropy on average.

If we look at specific languages and individual dependencies, the differences between the approaches can be quite striking, however. For instance, English nominal modifiers are often genitive constructions, which occur both before and after the head noun, e.g. the emperor’s new dress and the new dress of the emperor. Their position is strictly determined by the type of the genitive construction: the Saxon genitive always precedes the noun, and the Norman genitive always follows it. This is why English nominal modifiers have much lower entropy when the wordforms are taken into account than when the wordforms are ignored: 0.024 vs. 0.524.

The highest discrepancy between abstract and lexically-specific entropy is observed for Mandarin adpositions (0.99 vs. 0.01). They can be postpositions, prepositions, or elements of circumpositions, but their place with regard to the head noun depends very strictly on the lexeme. For example, the locative marker zài ‘at, in’ and the directional marker dào ‘to’ are always used before the noun, whereas the possessive markers de and zhī are used after the noun. Next come numeral modifiers in Basque (0.93 vs. 0.19), where the numeral bat ‘one’ in different forms, unlike all other cardinal numerals, takes the postnominal position (Hualde & de Urbina 2003: 118). This is a highly frequent numeral, which is also the indefinite article with the meaning ‘a certain’. It is difficult to distinguish between the functions in the corpus data. Similarly, Indonesian contains different types of determiners (0.87 vs. 0.29). Quantifiers like semua ‘all’ and beberapa ‘some’ are used before the head noun, while possessive and demonstrative pronouns follow the head noun. The smallest differences are observed in those cases where the entropy based on abstract dependencies is close to 0.

There are also some cases where the lexically specific entropy is in fact higher than the entropy based on the abstract dependencies, although the difference is usually very small. For example, English pronominal subjects have on average slightly higher lexically specific entropy due to the pronouns something and nothing, which are often used in the construction with the presentative adverb there, e.g. there is something in the air tonight. However, this variation nearly disappears when the entropy is computed at the level of the abstract dependency because of the high-frequency personal pronouns, which exhibit almost no variability. Such discrepancies are also observed in the other pronominal arguments (subjects and modifiers of nouns) and in adpositions in some languages. Thus, the presence of high-frequency exemplars with low variability tends to decrease the entropy on the level of abstract dependencies, whereas the presence of functional subcategories is responsible for its increase.

From all this follows that the lexically ‘naïve’ estimates of entropy give a reliable idea of the general magnitude of word order variation in a language, although they tend to overestimate the variation. This works only when many dependencies are taken, however. If one performs a classification that involves a small selection of word order patterns, he or she would be well advised to check the subcategories of dependencies and individual lexemes.

5.1 Causal factors that influence word order variation

If we want to explain the variation of word order patterns, several potential factors come to mind. One of them has already been discussed. Low word order entropy helps to disambiguate potentially confusable arguments. One should also mention the typical information-theoretic role of constituents. If a constituent frequently performs a particular role, which is associated with a specific position in discourse, then the position of the constituent will also be fixed. For example, constituents expressing topical, non-surprising and given information tend to have a fixed position in discourse. They occur either before the verb, close to the left, as in Mandarin or Russian, or after the verb, more to the right, as in Ute or Early Biblical Hebrew (Givón 1984: Section 6.5). Since subjects are often associated with givenness and topicality, especially transitive ones (Du Bois et al. 2003; Lambrecht 1994), they are expected to have a fixed position. We have seen already that pronominal subjects exhibit little cross-linguistic and intra-linguistic variation in their position with regard to the predicate (see Section 3.3). This can be explained by their topical role.

Another important factor seems to be grammaticalization, which limits the syntagmatic variability of linguistic units, i.e. the ease with which a word can be shifted around in context (Lehmann 2015: 167). For example, full verbs are more flexible than auxiliaries with regard to the verbal phrase with which they combine. Compare the verb ‘have’ in Classical Latin and Italian. In Classical Latin, the parts of the construction epistulam scriptam habeo ‘I have a letter written’ could appear in any order. Compare this with Italian ho scritto una lettera ‘I have written a letter’, where the auxiliary always precedes the main verb (Lehmann 2015: 168). From this follows that more grammaticalized functional elements should have more rigid positions with regard to their heads (i.e. nouns for adpositions, main verbs for auxiliaries) than less grammaticalized ones (e.g. nominal arguments or adjuncts).

However, grammaticalization itself is only an umbrella term for several different processes; it is not a functional pressure on its own. So, why are more grammaticalized units more fixed positionally? This can be explained by the fact that more predictable units have higher chances of grammaticalization to begin with. Units are more predictable when they often co-occur together with other units and become elements of highly entrenched chunks (cf. Bybee 2010). If these lexically specific chunks overlap with other chunks (e.g. my car, my book, her book, her phone, this phone, etc.), the pattern will be schematized as an abstract construction (e.g. [Determiner + Noun]), which can serve as the basis for emergence of syntactic constituents (Bybee 2002). Therefore, in order to obtain an abstract syntactic dependency with low entropy, one needs high frequency of two types: high token frequency of lexically specific chunks, and high type frequency of those chunks. On the other hand, grammaticalization (due to such processes as obligatorification and semantic bleaching) also boosts the frequency of the corresponding chunks and more abstract schemata, which leads to further increase in the relative frequency of the dominant word order and entropy reduction.

Next, one should mention processing constraints. In particular, the length of constituents has been shown to play an important role in word order. The preference to put long phrases after short ones was observed already in the classical rhetorical tradition (cf. Behaghel 1909/1910: 137–138). Behaghel (1909/1910) provided a first systematic account based on texts in Greek, Latin and German and called this preference Das Gesetz der wachsenden Glieder (the law of the growing elements). According to Hawkins (1994, 2014), human parsers prefer short constituent recognition domains. In a nutshell, these domains are minimized when shorter constituents, which are faster to process, are placed closer to the head than longer ones. This principle is known as “Minimize Domains”, or Early Immediate Constituents. Another principle is “Maximize Online Processing”, which helps to avoid garden path effects and delays in online property assignment. All this can be important for word order variability. For instance, VO languages tend to minimize domains and maximize online processing by putting complement clauses after the predicate. At the same time, the position of shorter dependent elements (e.g. one-word adjectives) is less crucial for domain minimization (Hawkins 2014: 101). This may affect the potential for variability of the corresponding word order patterns.

These factors and principles can be in conflict. For instance, the position of short elements is less important for the principle “Minimize Domains”. This will lead to high variability of their position before or after the head in comparison with long constituents. At the same time, short elements can also be highly frequent and grammaticalized, which means that their position will be more fixed. As was shown in Section 3.3, short functional elements (e,g. adpositions, auxiliaries and subordinators) usually have a rigid position in a language, as well as long clauses (with the exception of adverbial clauses). The position of mid-length constituents is usually more flexible. In the remaining part of this section, I will present a case study where these opposing factors are tested on corpus data.

5.2 Data and variables

In order to test the ideas discussed in the previous section, I used the data with the head-dependent patterns in the UD corpora (see Section 3.1 for more details). The entropy of a specific word order in an individual UD corpus was treated as the response variable. There were three categorical predictors, which are described below.

The first predictor described the functions of the dependent elements, which were classified into four categories:

1. function elements (adpositions, subordinators, auxiliaries, copulas, determiners);

2. core arguments and functionally similar clauses (subjects, objects, complement and subject clauses);

3. obliques and adverbials (oblique nominal and pronominal phrases, adverbial modifiers, adverbial clauses); ^[10]

4. modifiers of nouns and adjectives in a nominal phrase, with the exception of determiners (adjectival modifiers, attributive clauses, numerals and nominal modifiers of nouns).

Following the considerations discussed in the previous section, I expected function words to exhibit the lowest entropy due to their high level of entrenchment in the combination with their heads. I also expected adverbials and non-argument obliques to have the highest entropy, due to their multifunctionality.

Heaviness was operationalized in a binary way, whether the dependent element was a clause or not. This has to do with the fact that the lengths in orthographic words are difficult to compare cross-linguistically because the word is a problematic comparative concept (see the discussion in Section 6). At the same time, it is uncontroversial that clausal constituents represent the heaviest elements cross-linguistically. Their position is determined by the general processing principles (Hawkins 1994, Hawkins 2014), which were mentioned above.

Importantly, the previous studies show that the effect of length depends on whether the language is predominantly OV or VO. In order to take that into account, I created three categories, based on the proportion of OV in the language. The distribution is strongly bimodal, so it was useful to represent it as a categorical variable. The first category was “VO”, with the proportions of nominal objects followed by predicates ranging from 0 to 20%. The second category was “Flexible”, with the proportions from 20 to 80%. Note that this intermediate category was biased towards the languages with a mild preference towards VO. The third category was “OV”, with the proportions from 80 to 100%.

5.3 Mixed-effects regression analysis

This section reports the main results of fitting the linear mixed model with the variables discussed in the previous section. ^[11] I tested the pairwise interactions between the function and the OV/VO order, and between heaviness and the OV/VO order, in order to check whether the effects of the function and length vary in OV and VO languages. ^[12] These interactions are highly significant. The language, genus and dependency type were tested as random intercepts. Only the language and dependency type proved to be useful, according to the likelihood ratio tests. The table of coefficients and other information are provided in the Supplementary Materials: Regression Output.

Figure 7 shows the effect of the functional type of dependencies on the entropy for OV, VO and flexible word order. Not surprisingly, the flexible languages have the highest entropy values across core arguments, adverbials/obliques and nominal phrase elements, whereas the OV languages have the lowest values. Adverbials/obliques have the highest entropy, whereas function words have the lowest entropy, also in the OV languages. In the flexible and VO languages, we observe the following order: adverbials/obliques with the highest entropy are followed by core arguments, then by nominal phrase elements and finally by function words.

Figure 7:

Interaction between the function type of dependencies and the order of O and V. The bars represent 95% confidence intervals.

A post-hoc analysis of all pairs of estimates (see the Supplementary Materials: Regression Output) shows that there are no statistically significant differences between the functions in the OV languages. In the VO and flexible languages, function words are significantly different from adverbials/obliques and core arguments, but not from NP elements. Core arguments, adverbials/obliques and NP elements are significantly more variable in the VO and flexible languages than in the OV languages. There are no differences between function words in the three language types.

Next, let us interpret the interaction between the heaviness (clause vs. non-clause) and the order of object and predicate, which is displayed in Figure 8. The plot suggests that the heaviness effect is observed only in the VO and flexible languages. The clauses are indeed less variable than non-clauses, although they still exhibit substantial variation. In the OV languages, we observe a reversed order: the clauses are slightly more variable than non-clauses, but the difference is very small. The post-hoc tests indicate that the differences between clauses and non-clauses are not statistically significant. At the same time, the interaction in general is statistically significant and should therefore be kept in the model. This means that the effect of heaviness indeed varies across the languages with different OV/VO orders.

Figure 8:

Interaction between heaviness and the order of O and V. The bars represent 95% confidence intervals.

5.4 Interpretation of the results

We can draw the following conclusions. First, not surprisingly, the languages without a clear preference for OV or VO also exhibit the highest entropy in all conditions. The predominantly OV languages display the lowest entropy.

Second, function words have the lowest entropy in all languages, and adverbials/obliques have the highest, although the differences are statistically significant only in the VO and flexible languages. Therefore, entrenchment due to high frequency can make a difference, provided there is enough variability in a language. The adverbials/obliques have the highest entropy, most probably, due to their multifunctionality.

Third, there is a significant difference in the heaviness effect on entropy between the OV languages and the VO and flexible languages. There is a tendency in the latter to have lower entropy of clauses in comparison with non-clauses, whereas the OV languages display hardly any difference. This supports the processing-related explanation, which was mentioned in Section 5.1. According to Hawkins (1994, 2014), object clauses with the subordinator in the initial position, as is the case in VO languages, tend to follow the main clause. This order makes the processing optimal based on both principles, “Minimize Domains”, and “Maximize Online Processing”. As for OV languages, these principles are in conflict. Therefore, there is no perfect solution, although typological data suggest that the postverbal position is the more popular one (Schmidtke-Bode & Diessel 2017). The same logic applies to relative clauses (Hawkins 2014: Ch. 7), which are usually postnominal in VO languages, but can be either postnominal or prenominal in OV languages.

Importantly, Hawkins’ principles were formulated on the level of languages and their preferred strategies. The regression model shows that the differences in the effect of heaviness between the VO and VO languages are discernible also at the level of usage tokens and probabilistic tendencies. ^[13]