Automatic classification of older electronic texts into the Universal Decimal Classification–UDC

1.1 Problem description

Like other digital libraries (e.g. Europeana, Open Library (www.openlibrary.org), Library of Congress (www.loc.gov/collections/) and others), there are vast numbers of free digital resources in the web. In the Republic of Slovenia, one of the richest and most complete free electronic sources is the Digital Library of Slovenia (www.dlib.si) containing more than 850,000 electronic (digitised and digital) publications. Digitised sources are those that were originally only printed and later transformed into a digital format, whereas digital sources are those that are originally created and made accessible in digital format (and may also be printed). In the following text, we will use the term “digital publications” for all written sources available through the Digital Library of Slovenia. Usually, items included in a library would have a bibliographical record. Scholarly publications are systematically bibliographically processed, which means they have a bibliographical record in the library catalogue, and therefore one or more classification numbers from the UDC system. In contrast, most older texts and sources have not been bibliographically processed (e.g. articles and texts from older printed magazines and newspapers from the field of culture) and therefore are not classified into UDC system.

On the UDC Consortium's website, the classification is described as one of the first universal classification systems and remains one of the most widely used international classification systems in librarianship. It was developed at the end of the 19th century by the Universal Bibliographic Repertory, based on the Dewey Decimal Classification (DDC; 1876), by Melvin Dewey (Dale, 1978; Kendall, 2014). Other well-known systems are also the Library of Congress Classification (LCC), Colon Classification (CC) and Bibliographic Classification (BC) (Miksa, 2017). The UDC classification is based primarily on the numerical (“decimal”) labelling of the contents of articles with line item sequencers in the form of a sequence of numbers and symbols; it is universally portable, highly structured, the notation is precisely representative of the subject content and understandable between languages. It is built so that it can be expanded and upgraded with new classifications. The classification is used worldwide and is currently translated into more than 50 languages. According to Slavic (2008), it is the second most used classification in the world. It allows unlimited assembly of classification attributes (e.g. master table, place, time, etc.) and relations between them to describe the subject (in our case) of the publication. Because publications in the union library catalogue (the COBIB union bibliographic/catalogue database) in Slovenia use the UDC classification scheme, we used this one. It contains nine basic groups:

1. Science and knowledge. Organisation. Computer science. Information. Documentation. Librarianship. Institutions. Publications

2. Philosophy. Psychology

3. Religion. Theology

4. Social sciences

5. Mathematics. Natural sciences

6. Applied sciences. Medicine. Technology

7. The arts. Recreation. Entertainment. Sport

8. Language. Linguistics. Literature

9. Geography. Biography. History

Group 4 is empty. It originally consisted of linguistics, which was later transferred to Group 8.

An example of the classification of a record in a library catalogue can be made for Ivan Cankar's drama, “Hlapci”. In the COBIB (union library catalogue), the following UDC numbers are identified for it:

821.163.6-2 (Slovenian literature, dramatics) 821.163.6.09 (Slovenian literature, literary publication)

For newspapers, they are usually classified as 070 –“Newspapers. Printing. Journalism” or 050 –“Serials. Periodicals”.

On the website of the Digital Library of Slovenia, it is possible to search the content of the articles only through the full text. It is currently the best tool for discovering older texts. However, using and researching articles and other publications in such a way only does not offer good user experience, due to optical recognition deficiencies (poor quality of text recognition in newspapers and serials of the older type, use of old Slovene script like “metelčica”, “dajnčica”, “bohoričica”, “gajica”, etc.) and too many returned search results. For the majority of the texts and copies of serials, there is only one bibliographic record in the library catalogue. Examples of this include “The Laibacher Zeitung”, a newspaper, with more than 58,000 issues and many more articles, Ljubljanski zvon-“The Bell of Ljubljana”, with more than 11,000 articles, or Dom in svet-“The Home and World” with over 16,000 articles, etc. The easiest way to illustrate the present situation is the following example: all the articles of the serial “The Home and World” that originated between 1888 and 1944 are placed in the UDC classification sub-group Slovene Literature and Culture (821,163.6 and 008) which means that every article within the magazine “Home and World” is classified as “Slovene Literature” and “Culture” and nothing more than that. If we mention some well-known magazines from those times and describe their content, these would be: “Home and World”–it was a Slovenian literary monthly, which was created as an entertaining and educational magazine for Roman Catholic readers and later developed into a literary magazine. It was founded by the philosopher and theologian Frančišek Lampe and edited until his death (1900). The magazine was initially distinctly Catholic but represented a more tolerant and, above all, the most artistically creative part of Catholic culture. Another one is “The Bell of Ljubljana”–the central Slovenian literary newspaper was published monthly. In addition to literature, it also contained art criticism as well as discussions and essays on the arts. At first, it was more scientifically oriented but later limited to the humanities, and from 1931 also with articles on current social issues. We can also mention “Agricultural and handicraft news”; it was first intended to help farmers and artisans, but later also carried articles in the fields of literature, conservative politics, culture and correspondence from various places. They were important mainly for the consolidation of the Slovene literary language, the general acceptance of the “gajica” and, in general, for the all-round cultural development of the Slovene nation. Gajica is a Latin script developed by the Croatian linguist Ljudevit Gaj. It was first used to write Croatian, but later, with some adaptations, it was also used to write Slovene.

3.1 Data preparation

At the core of the design cycle, as described by Hevner et al. (2004), is the classification model building process. The research process of data collection and data analysis methods, which are presented in Figure 1, are described in detail as follows.

We exported the attributes described below for each text into a .json file to be further processed from the MSSQL database in which we store the data. In total, we exported more than 70,000 scholarly articles and more than 200,000 old texts. In order to uniquely distinguish content and input attributes for model construction, the database included the “Title”, “Full-text of article”, “Identifier” and “UDC numbers”. For the old texts, the UDC numbers were those defined for whole magazines, journals or newspapers, as already stated.

This phase can also be called the “pre-processing” of data. We cleaned out the words that were not useful and only introduced stop words into the model. It is a process of cleaning the text (deleting non-alphanumeric characters, blank lines, etc.).

The process of text classification model building starts with preparing a corpus of texts. In our case, we have two corpora: the first corpus of more than 70,000 scholarly texts that are all bibliographically processed, meaning they are assigned with UDC number or numbers by a human expert. The second corpus consists of 200,000 old and bibliographically not processed texts (articles, short notices, popular texts) that were mostly published between 1850 and 2000 in journals and newspapers in the Slovene language used at that time. Slovene is morphologically one of the more difficult/rich languages and, in its history, it has had several variations (i.e. bohoričica, meteljčica, dajnčica) which makes it difficult to compare with language in scholarly articles written in the present day. Consequently, we had to pay extra attention when cleaning the texts, using “lemmatisation”–acquiring the base of the words and replace old vocabulary with that used today. We used two approaches in the cleaning process, as well as one already built model–FastText.

1. Minimal processing – MP (we only retained words that were alphabetic (containing only alphabetical characters) and are lemmatised using a file that contains the root form (lemma) and all the words that share this root. If the word does not exist in the root dictionary, we left it in its original form.)

2. Regular processing–RP (similar as above, plus we also removed “stop words” and words that we do not want to include in the list of words; mostly irrelevant words). We also removed all words smaller than three characters and those whose root or lemma was not in our dictionary.

3. FastText–FT. The FastText model can be seen as a shallow neural network that derives its capabilities by scaling up the number of learnable vector embeddings of n-gram features that are fed into the network (Agibetov et al., 2018). Its database contains data from Facebook and is translated into more than 150 languages.

The ML phase can be done with different programming languages, such as Python, R and similar. Like Jimenez-Marquez et al. (2019), we used the NTKL (Natural Language Toolkit) with Python, which offers easy-to-use interfaces and language resources, such as WordNet, along with a collection of word-processing libraries for sorting, tokenisation, perception, marking, parsing, semantic reasoning and similar (Bird et al., 2009).

3.2 Structuring

First, the vocabulary had to be “separated” into tokens or words. This process is called “tokenisation”, which is a step in a process which that longer strings of text into smaller pieces, called “tokens”. Larger pieces of text can be tokenised into sentences and sentences into words. Tokenisation is enabled in Python programming language using the NLTK library and the word_tokenize function (Jimenez-Marquez et al., 2019; Ramdass and Seshasai, 2009). After that, we apply lemmatisation to obtain the dictionary form of words. For the old text corpus, we implemented another step: synonyms replacement. Since the old corpora also use anachronisms, we had to “translate” them into their current form. We used a dictionary with old words and lemmas to convert old words into their current forms. To make tokens useful for the classification, we had to convert them into numbers, which is the process of vectorisation. Similar to the work of Farkas et al. (2010) and Zhang et al. (2016), we used the Tf-idf method to model the matrix of word vectors appearing in texts.

When building feature vectors from texts, we did not use term frequency but inverse document frequency. According to Aggarwal and Zhai (2012), in general, a common representation used for text processing is the vector-space based TF-IDF (term frequency-inverse document frequency) representation. In the TF-IDF representation, the term frequency for each word is normalised by the IDF. The IDF normalisation reduces the weight of terms, which occur more frequently in the collection, which reduces the importance of common terms in the collection, ensuring that the matching of texts is more influenced by that of more discriminative words that have relatively low frequencies in the collection. The result is sometimes called “the semantic vector” (Jalil et al., 2016). The result of vectorisation is the mxn array, which stores a dictionary of words appearing in texts. After the vectorisation step, the vocabulary (an array with vectors representing the articles in n-dimensional space) was ready for the model implementation.

3.3 Model implementation

In this step, we had to perform three key tasks: splitting the data set to train and test set, selecting an algorithm and fitting the model. For this purpose, we split the data set, consisting of more than 70,000 scholarly texts, to train and test subsets in the ratio of 80/20 (57,039 instances used for training the classifier, and 14,299 instances used for testing the classifier). After the step of dividing the data set into learning and testing, we used algorithms to build ML models. Each algorithm built its model on the training data set, which was later tested with test data.

The data set of old texts consisted of more than 200,000 articles. Validation of the performance of the classification model (Figure 1, fifth step) was done by human experts: 15 librarians who assessed the automatic UDC classifier on 150 randomly selected texts, since they were never classified by librarians.

In the research, among the algorithms of supervised learning of which are many (e.g. Bayes classifier, Decision trees, Support vector method, Neural networks, Genetic algorithms, Latent semantic analysis, k-nearest neighbours, etc.) (Abdul et al., 2015; Baharudin et al., 2010; Bhalla and Kumar, 2016; Chowdhury et al., 2019; Du, 2017), we chose Linear Classifiers (Logistic Regression (LR) (Healthy and Survey, 2014; Ng and Jordan, 2001), Naive Bayes Classifier (NB) (Asy'arie and Pribadi, 2009; Kononenko, 1993)), Support Vector Machines, (SVM), (Bhalla and Kumar, 2016; Cortes and Vapnik, 1995), Kernel Estimation (K-Nearest Neighbours, (k-NN) (Baharudin et al., 2010; Rawat and Choubey, 2016)), Neural Network (Multilayer Perceptron (MLP) (Collobert and Bengio, 2004; Na et al., 2019)).

ML algorithms are described as learning a target function (f) that best maps an input variable (X) to an output variable (Y): Y = f (X). The main goal of ML is to learn the mapping Y = f (X) of Y prediction for a new input X. This can be called predictive modelling, for which the goal is to obtain the most accurate prediction possible. Fitting the model is making the algorithm learn the relationship between predictors and outcome so that prediction for the future values of the outcome is possible.

To assess the performance of the trained text classification model, for which the target variable can have two or more classes, the measures of Precision, Recall, and F1 score are used. The following Equations (1)–(4) are required to determine the metric values of the confusion matrix (Abdelaziz et al., 2018; Joo et al., 2013).

3.4 Model verification

Normally supervised learning techniques are used for automatic text classification, in which pre-defined category labels are assigned to articles based on the likelihood suggested by a training set of labelled documents (Baharudin et al., 2010). In supervised ML, our task was divided into two phases. Learning and testing 70,000 scholarly articles (UDC was available in the bibliographic catalogue) and use the trained model to classify 200,000 old non-scholarly articles. In the first phase, we tested 20% of the scholarly texts (test set), which amounted to 14,299 articles. We used classification accuracy (CA), Recall, Precision and F1 measures. The following Equations (1)–(4) are required to determine the metric values (Abdelaziz et al., 2018; Joo et al., 2013).

1. Accuracy–the ratio of correctly classified cases to all cases of the observed set.

2. Precision–the ratio of correctly classified positive cases to all positively classified cases of the observed set.

3. Recall–the ratio of correctly classified positive cases to all positive cases of the observed set.

4. F1 score–harmonic mean of precision and recall.

4.1 Clustering of the scholarly articles data set

In the bibliographic catalogue, all the articles are classified with the usage of UDC; therefore, it was straightforward to check whether the naturally occurring clusters in the scholarly corpus are aligned with the assigned UDC class. Figure 2 shows articles in 73 different clusters (articles represented by dots, colours representing clusters) (abbreviated to the first two digits of UDC, there were 73 different UDC in 900 articles). Articles with similar content tend to be closer to one another and thus form clusters, which are graphically represented by dots in 3D space.

Zoomed-in detail on the right of Figure 2 shows the cluster (or a group), one of the 73 groups. This is an example of a “clean” group, since elements tend to stick together. In this group, the articles contain content about Christianity (UDC = 27). Most significant (TF-IDF) words from articles in this group are “Church, man, life, God, faith, council, God's, question, etc”. More than 90% of all articles/elements in this group have UDC = 27 (in the bibliographic catalogue). Another example of a very clean group is the one in which the articles have UDC 82 (literature). Most calculated significant words in this group are: “song, time, sky, eyes, heart, Earth, sun, wind, water, children, love, night, etc.”.

We used the clustering method to explore the input data and assess whether we can use this data to conduct the classification training. In so doing, we checked the relationship between publications and their UDC rows on the one hand and clusters on the other on the operation of the k-means algorithm. As can be seen in the example on the right side of Figure 2, clusters were homogeneous. Also, it is true that each publication can have more than one UDC number and should therefore appear in multiple clusters (there is high relation between religion and architecture, so the article should be in both clusters, but it is only in one). The disadvantage of using only unsupervised learning methods is that the system operates only with unmarked data during learning and has no insight into the correctness of the results; its objective is to identify hidden structures in unlabelled data (Vakharia et al., 2015). The use of unsupervised learning methods on a small set of articles (900) served in a research analysis.

4.2 Classification model building for scholarly texts

The complete corpus of 70,000 and more scholarly articles was divided into training and testing sets, 80% instances used for training (57,039 texts) and the remaining 20% instances used for testing (14,299 texts). Hypothesis 1 states that it is possible to build such a classification model on the corpus of scholarly texts, which would assign any randomly chosen publication from the test set to at least one appropriate UDC group with the probability of at least 0.8; it was tested using CA, Recall, Precision and F1 measures. The results of classification algorithms performance on the test set of scholarly texts are presented in Table 1.

The results obtained for the classification of scholarly texts corroborate Hypothesis 1. As evident from Table 1, at least 80% of articles are accurately classified into an appropriate UDC group. The best performing classifier, according to the classification algorithm, is SVM using Tf-idf (CA = 0.963). When the number of features (i.e. individual measurable characteristics of a subject being observed) is low, SVM, LogReg and MLP algorithms perform better than NB or KNN (Colas and Brazdil, 2006; Musa, 2013; Zanaty, 2012).

4.3 Using the classification model to classify old texts

Next, we tested Hypothesis 2, stating that the classification model trained on the corpus of scholarly texts can be used to classify the corpus of old texts. The old texts are not fully bibliographically processed (texts are merely assigned the UDC of the parent's bibliographic record–magazine or newspaper), often poorly structured, written in archaic Slovene language and vary in length compared to the scholarly texts. The vocabulary of the language in the 19th and early 20th centuries, compared to the present-day language is quite different because the vocabulary itself and language of each nation are changing over time. The complexity of the sentences and the choice of words are also different than in the academic literature. By reviewing the article, the librarian can give an opinion on the correctness of the classification given by a classification model.

Therefore, the main challenges of this task were the fact the language that had changed significantly within one century and the fact that the principle of writing scholarly texts is considerably different from the writing of popular texts. For this reason, we used a dictionary and translated old Slovene words (where possible) into the language used today, in order to make the classification algorithms work better. If we had a large corpus of bibliographically processed old texts, we would have used it for a learning set and probably would have achieved even better results.

Once we had confirmed Hypothesis 1, we used the complete corpus of scholarly texts (more than 70,000) as a training set, following the basic idea that the classification model's performance is better with a larger set. We then used this classification model to classify the 200,000 old texts. The trained classification models served us determining UDC for old texts.

The algorithms have placed the articles in one or more different UDCs. For further analysis, we considered only UDC numbers with at least 10% probability to fit in some UDC group by the classifier and sorted them by probability, for example, “KNN: [(“821”, 0.59), (“7”, 0.19), (“929”, 0.12), (“111”, 0.1)], where the first number in the bracket is the UDC group, and the second is the probability of correct placement in a group, in this example calculated by KNN algorithm.

A total of 150 randomly selected articles, automatically assigned with UDC classes, were evaluated by 15 librarians (each librarian had the task of evaluating 10 randomly assigned texts). The text and the results of the automatic classification according to three different text processing and vectorisation approaches (minimal processing, regular processing and FastText) were available for each of five classifiers (Naive Bayes classifier, Support vector machines, Multilayer perceptron, Logistic regression and k-nearest neighbours algorithm).

For better understanding the work of the human experts, we present a few examples below. Librarians labelled calculated UDC numbers with green if the proposed UDC by the classifier is appropriate and with yellow if the proposed UDC by the classifier is appropriate in a broader context. Proposed UDC numbers that were not labelled were not appropriate for the article that was processed. An important difference between UDC numbers assigned by librarians (for the entire journal) and the articles reviewed is that the classifiers mostly “found” and “suggested” UDC numbers that described the article by content and not only type (e.g. 070 - Newspapers. The Press. Journalism, 050 - Serial publications, periodicals (as subject)). We explained this in more details by giving three examples.

Examples

Example 1: “Electric Robot” https://www.dlib.si/details/URN:NBN:SI:DOC-Y1E3CHY4 from the newspaper Slovenski Gospodar, volume 72, issue 40, year 1938. It was published mostly weekly in Maribor.

Electric robot (article, translation from Slovenian language): “In Czechoslovakia, a mechanical device was introduced, which automatically regulates the lighting and switch-off of the light. The robot consists of two cells. The first one is in the building of the city power plant, and the other on the transformer, as soon as it is in the evening, it reacts both cells to the change of light by burning everywhere electric light.”

Since the article is very short, it seems important not to discard too many words from the corpus. In Tables 2 and 3, we display the results of automatic classifiers where minimal and regular processing was done. In Table 4, we display the results of an article that was processed with FastText.

UDC numbers for the entire publication (from the record in the library catalogue) are:

1. 070 Magazines. Print. Journalism

2. (497.4) Slovenia. Republic of Slovenia

3. “18/19” 19/20. century

Calculated/suggested UDK numbers accepted by the librarians:

1. 007 Activity and organisation. Communication and control theory in general (cybernetics). “Human Engineering”

2. 53 Physics

3. 620 Material Investigation. Blessing. Power plants. Energy

4. 621 Mechanical Engineering General. Nuclear technology. Electrical engineering. Mechanical technology in general

5. 681 Precision mechanisms and instruments

Interestingly, using FastText, we also obtained highly ranked UDK numbers:

1. 53 Physics

2. 537 Electricity. Magnetism. Electromagnetism

3. 628 Health Tech. Water. Sanitary appliances. Lighting technique

According to librarians, physics (53) is too a broad term, but UDC 537 has been confirmed.

Not appropriate calculated UDC numbers were

1. 331 Statistics as a science. Statistical theory (p = 0.19)

2. 34 Law. Jurisprudence (p = 0.96)

3. 544 Physical chemistry (p = 0.19)

4. 57 Biological sciences in general (p = 0.26)

5. 579 Microbiology (p = 0.11)

6. 616 Pathology. Clinical medicine (p_max = 0.02)

7. 628 Public health engineering. Water. Sanitation. Illuminating engineering (p = 0.78)

8. 638 Keeping, breeding and management of insects and other arthropods (p = 0.03)

9. 669 Metallurgy (p = 0.06)

10. 821 Literatures of individual languages and language families (p_max = 0.18)

11. 930 Science of history. Historiography (p = 0.01)

Except in some cases, the algorithms have labelled mostly irregular UDC classes as suitable with low probability. If set put a condition (e.g. the minimum probability of placement should be at least p = 0.2), most of them would drop out considerably, and librarians would have less work to do with validating appropriate rows. In contrast, UDC 621 was suggested with the following probabilities: [0.15, 0.23, 0.29, 0.40, 0.46, 0.52, 0.57, 0.59, 0.71, 0.90, 0.99, 0.99] which gives us an average of p = 0.57 and seems more credible than probabilities below p = 0.2.

The classifiers perform surprisingly well in old Slovenian texts. An example from “Kmetijske in rokodelske novice” shows us so. Agricultural and handicraft news was published weekly by Janez Bleiweis. They were first intended to help farmers and craftsmen, and later published articles in the fields of fiction, conservative politics, culture and letters from various places. They were important mainly because of the consolidation of the Slovene literary language, the general acceptance of the “gajica” language and, in general, the comprehensive cultural development of the Slovenian nation. The newspaper continues under the headline “Novice kmetijskih, rokodelnih in narodskih reči”.

For example, we took a very short text whose contents are shown in Figure 3.

Example 2: Krajnski Vertnar, https://www.dlib.si/details/URN:NBN:SI:DOC-TYG137JU, from the Journal “Kmetijske in rokodelske novice”, Volume 1, Number 1, Year 1843.

Forecast of agricultural books (article/notice, translation from old Slovenian language):

“Forecast of agricultural books, For sale in Ljubljana at the bookstore Mr Lerchar on the big square: Krajnski Vrtnar, or teaching to grow many fruit trees in a short time, to ennoble them by grafting, and plant beautiful gardens with great benefit. The Imperial Royal Society of Agriculture in Carniola was brought to light. Written by Franz Pirc, the pastor at Sv. Jernej in Loče. In Ljubljana 1834–1835. Price 24 crowns.”

Tables 5–7 show the calculations of the classification results for the text “Krajnski Vertnar”

UDC numbers for the entire publication (from the record in the library catalogue) are:

1. 070.48 Specific types of newspapers

2. (497.4) Slovenia. Republic of Slovenia

3. “18” 19th century

Calculated/suggested UDK numbers accepted by the librarians:

1. 655 Graphic industry. Printing. Publishing. Bookbinding

2. 821 Literature of individual languages and language families

3. 908 Examining Areas. Exploring Places [Home Science]

4. 39 Cultural Anthropology. Ethnography. Customs. Habits. Tradition. Lifestyle

5. 398 Folklore in the narrow sense

We were interested in the work of classifiers on a short, unrelated, unstructured text. The following is an example from Kmetijske in rokodelske novice (Agricultural and Craft News), the content is shown in Figure 4, and the results of classifiers in Table 8.

1. 633 Field crops and their production

2. 339 Trade. Commerce. International economic relations. World economy

Even if the text was very short and unstructured, we still obtained two good UDC numbers, which are a description of the content and not only (as it is in the catalogue for the whole newspaper–070: Newspapers. The Press. Journalism) by type of publication.

The examples shown here indicate the power of automatic, (actually semi-automatic since validation work is still required) of the classification. It is interesting that even though in some cases these are relatively short texts, we received validated UDC numbers suggestions from different main UDC groups: 0, 3, 6, 8.

In Figure 5, we report the analysis of human experts (librarians) of the evaluation of the 150 randomly selected texts. As seen, every bar displays an article tested by librarians. The green bar above the identifier of an article presents the count of appropriate UDC number or numbers, confirmed by the human expert (librarian). The yellow bar presents an appropriate UDC number in a broader context, as stated by librarians. As we can see in Figure 5, most of the articles were approved with at least one UDC number by a librarian. We should take into consideration that it is not necessary that every librarian would label or choose the same UDC numbers for the same articles. Even in real life, in the cataloguing process, classification by librarians varies. In the research by Marijan and Leskovar (2015), they showed that the work that includes human decision-making could vary from decision-maker to decision-maker. Librarians are independent in their work, in determining the UDC numbers, in the process of cataloguing. For this reason, they reviewed and assigned UDC numbers for different sets of texts in the present study.

The calculation of the average appropriate or appropriate UDC numbers in a broader context for 150 articles is:

1. On average 1.8 approved UDC numbers per text were confirmed by librarians

2. On average 2.55 approved or approved in broader context UDC numbers per text were approved by librarians

3. For four texts, only UDC numbers approved in broader contexts were accepted by librarians

4. The minimum value for approved UDC per article was 0 (in four cases of articles/texts)

5. The maximum value for approved UDC per article was 6 (in one case of article/text)