1 Introduction

Fragments or whole objects buried in the ground can be found by anyone. If we lack knowledge about it (e.g., historical age and conditions), we may not be able to take decisions, especially important for any cultural heritage. Lack of knowledge becomes the reason why important elements of our past might have been lost forever. To remedy this, we can educate society about culture and history, what would be reflecting in discovered artifacts from ancient times. Of course, solely teaching the public will not necessarily be practical, because estimating the state as well the features of a given object can be a difficult task. Through the ages, humans made various constructions and used various technologies which resulted in a wide variety of artifacts due to climate, material, style, and purpose which makes estimation of excavations even harder (Fig. 1a–f) and Fig. 2. In addition, gained knowledge often may not be sufficient. The solution is introduction of latest technology into the process, both for better and faster evaluation of potential historical values of items from excavations.

Fig. 1
figure 1

Sample excavation sites (a–c), churches (d), and museums from various periods of the history (e–f) starting from classical and late antiquity, through Middle Ages, renaissance, until early modern. Each of the places differ in climate, environmental conditions, and also in culture what makes each of them unique excavation site

Full size image
Fig. 2
figure 2

In various excavation sites, many artifacts like sculptures or pottery are found, which, depending on the material of the work, the epoch from which they originate and the conditions in which they were until their discovery, need more or less professional care to classify and reconstruct them. In the above pictures, we can see artifacts from world famous places like Delphi in Greece related to Classical and Late Antiquity and artifacts from found in Malbork Castle in Poland related to Middle Ages

Full size image

Searching for information on the Internet can be a difficult challenge, which will involve interpretation of the object or its fragment by the anise of material, location in which it was found, and time estimation for when it was created. Almost every person has a mobile phone in which various sensors are built-in. The most desirable can be a camera which can easily and quickly take a photo of the object, then by the use of intelligent methodology analyze and classify it. Such a mechanism can efficiently support decisions done by people to recall some of knowledge gained at schools over the years.

Excavation artifacts can be quickly analyzed by means of simple image and specific, dedicated technique of classification and inference. In this paper, we propose the operation of such a system and the technique of inference based on soft trees. The idea of soft trees is a hybrid solution through the combination of two mathematical structures such as soft set and decision tree. In addition, for the purpose of image processing, we suggest defining the operation of tree vertex by using devoted neural classifiers Fig. 4.

1.1 Related works

Computer techniques can support important decision processes. We can find many applications in various fields where images are used to carry information, for which convolutional neural networks (CNN) are used as object detectors and classifiers. In [14], a construction of deep spiking CNN was used to support recognition processes. Authors investigated various models and training algorithms to find optimal settings. Similarly in [5], an extensive study was presented for the optimal balance problem in CNN for complexity in image processing. Object recognition by CNN architectures adjusted to the model was discussed in [23].

Object recognition is not an easy task so the solution modeled to do it very often represents some complex or hybrid ideas. In [18], to use a composition of local extreme with learning machine was proposed, while in [16], to compose distance metrics with CNN for face recognition was proposed. Neural networks are efficient in texture differentiation for marine surface models as described in [25] and decision systems for high-speed trains [11]. In [17], a hybrid of cascade model composed with deep CNN was reported for efficiency in high-resolution images, while [3] discussed a composition with orthogonal planes for dynamic texture recognition applied in surveillance, medical imaging, and remote sensing. In [13], CNN with a support from adjusted coarse segmentation was used to help on magnetic resonance image processing for potential prostate detection, while in [1], CNN was used as filter for spam images in real time processing. A survey on advanced applications of CNN for recognition from color images can be found in [9]. Among important training methods of neural architectures, we shall consider RMSprop (Root Mean Square Propagation) which in various configurations was presented in [20] and [29].

Information systems very often serve as platforms for decision support from multimedia [28] where implemented mathematical theories simulate rules of decision-making. Recently many examples have been given for two of them. Soft sets represent a concept of simplified relations between decision classes. In [8], soft sets were presented as multi attribute theory for isolationistic decision-making. In [4], this concept was proven to handle uncertainties of multi criteria object description, in [21], it was shown that soft sets are also efficient in situations when information about input object is incomplete. Such models are very useful in real-life applications like medical diagnosis based on Dempster-Shafer evidence theory [26] or group decision-making processes [2]. Decision trees are mathematical structures naturally related to decision processes, since each node represents situation with possible choices. Some properties let these structures to get split for parallel data processing [12] or extract information from streams [22], while others make it possible to unify processes for instant item selection [7] or develop incremental decision structures [24]. There are various applications like emergency management [10] or even inbound ships management which can be done with human-like intelligence [27].

Archeology and excavations are very good examples of ubiquitous computing applications. In these science, technology and culture join their efforts to solve complex problems and answer questions which seems to have no response. In [6], how to use CNN as controllers for 3D models reconstructions of excavations artifacts was presented, while in [15], ground-penetrating radar image processing was used for localizing artifacts in excavation unit. Both technologies helped to reduce errors in evaluation of buried artifacts.

Our proposed technique is a novel proposition of hybrid decision support system, in which we unify decision trees with soft set concepts. Proposed novel decision model evaluates consecutive features of excavated artifacts on the way of simple comparisons between features, what we called a soft tree. In each node of the soft tree, we use CNN to evaluate different features of the artifact. In this way we compose a novel classifier able to help in excavations. The proposed technique shows effectiveness in solving described problems related to checking whether found object can be considered a heritage and to be called an archeological artifact.

2 Soft trees

Let us now introduce proposed novel decision system called soft tree—the idea of soft set and decision tree as a combination. This idea was modeled for classification problem under certain conditions of uncertainty.

2.1 Soft sets

The idea of soft sets was introduced by Dmitri Molodtsov as mathematical model for reasoning in conditions of certain uncertainty [19]. Let us assume that the class of considered objects will be marked as U, and as P(U), we mark a set of all subsets of U. In addition, the set of parameters will be marked as E, then the individual parameters will be subsets in E as \(A\subseteq E\). Using these assumptions, a soft set will be defined as a pair (F,A), where F is the assignment defined as follows

$$ F:A\to P(U). $$
(1)

Moreover, (G,B) ⊂ (F,A) will be called a subset if and only if B ⊂ A and ∀𝜖∈BG(𝜖) = F(𝜖).

Each soft set can be presented in the form of a binary table by defining additional subset Q ⊂ A called set reducer P with condition of independence of both sets, i.e., formally IND(Q) = IND(P). The selected inference consists of calculating the maximum sum of the products of selected ci which equals to the value of the soft set features hi ∈ U what can be presented as

$$ \max\limits_{i \in 1, {\dots} , I}{c_{i}}=\max{\sum\limits_{j=1}^{J}h_{i,j}}. $$
(2)

In general, if hi,j ∈ F(𝜖), we value it as hi,j = 1 and if hi,j∉F(𝜖), we value it as hi,j = 0, but also other values representing selected decision classes are possible.

2.2 Classic decision trees

Decision tree is a mathematical structure, i.e., a graph without cycles in which we have only one path between two nodes. Such structure allows to represent the division of a set into specific classes, more precisely the division of nodes is understood as a way of distinguishing decision classes. The tree begins with the root from which at least two edges extend to next elements. In addition, the roots from a given node to the next one form a level structure, where the addition of new edges leads to a lower one. Each node is associated with a specific question related to the value of the attribute. We assume that if a given object has a given feature, it goes down a level below the edge which corresponds to a given value. Nodes which do not have any edges on the lower level are called leaves and class representatives.

2.3 Soft decision trees

Unlike the classic tree, where there was class data in the leaves for the classification process, newly proposed soft tree does not have classes in this understanding. We understand a decision class as each level of the tree. Each node (representing evaluated feature) is equally important and is equivalent to a certain attribute describing the object. In addition, we assume no obligation for the attribute; hence, the value of the assignment of a given attribute depends on the single decision value, and thus range in 〈0,1〉.

We simply reduce a classical decision tree paths to consecutive levels in soft decision tree (see Fig. 3). After calculating all nodes, the obtained features are interpreted as a soft set. A given node in the soft tree corresponds to decision on one feature, so the whole level corresponds to various possibilities related to various features representing the same decision class. It means that for one decision, a result can be interpreted as few possible choices in one decision class. We find the best fit with all declared objects using classic inference model in soft sets described in (2). We assume that selected value of the node is considered as the maximum feature in decision class. Therefore, we can add/remove any feature to the decision class or even add/remove entire decision class. Such structure of decision-making process implements a great freedom for the final classifier, which evaluates subsequent levels of the soft tree without interruptions. Proposed process is presented in Algorithm 1.

figure a
Fig. 3
figure 3

Visualization of proposed soft tree reduction from classical decision tree. Each question is represented in the level of the proposed solution; therefore, it is very simple to extend it of any new feature for evaluation. The whole soft tree idea needs only additional nodes or levels in soft tree architecture with CNN trained for evaluation of this feature. This simplicity in construction and therefore possible extension is related to soft set decision classes, which are very simple to expand just by adding new categories

Full size image

In this way, we receive a novel decision support model—soft tree, which does not need complex decision rules and moreover it is not necessary to know each of attributes. The last problem we need to solve is proper classification of excavation images, what we have solved by the use of CNN.

3 Soft trees for image processing

For image classification, we redefine a node in the above soft tree idea. Input data will be understand as numerical value or image. Depending on the type of processed information, a different classifier will be applied. For the purposes of analyzing archeological artifacts, we use convolutional network trained for classification of specific features in each node. Proposed classifier enables reduced complexity of analysis, and contributes to reducing training time but also architecture of the network in which we will need only a small number of layers. The decision of CNN is placed in soft tree node, from which a full soft set decision table with set priorities is composed in the next stage of decision-making process. Sample visualization of the node is shown in Fig. 4. Each feature is analyzed on different level of the tree.

Fig. 4
figure 4

Visualization of vertex node in soft tree. Proposed solution is very simple for extension of any new feature evaluations. The whole idea needs only additional nodes in soft tree with CNN trained for evaluation of this feature, since soft set decision class is also very simple to expand just by adding new examples

Full size image

3.1 Convolutional Neural Network

Convolutional network is a mechanism of classification which receives a two–dimensional image as input. The image is treated as a three-dimensional object spread on three axes height, length, and depth. The third dimension depends on the color system which is being analyzed. The most commonly used model is RGB (red-green-blue), which defines three colors in the image; hence, the depth of the image will be equal to three. The incoming image is an interpreted asset of three matrices, where each element, that is, every pixel is treated as neuron. In addition, the network architecture involves the use of three types of layers. The first of these is convolution layer, which performs a convolution between image matrices and declared graphic filter. The next type is pooling layer which reduces the size of the image. The reduction is based on specific function, usually maximum. The specified function operates in the area of declared neighborhood. For a given grid, a pixel fulfilling given function is transferred to the next layer (to reduced image). The third and last type is fully connected layer, which resembles classical neural network composed of neuron columns, where the first of them is composed of n neurons. The n number relates to all pixels in the previous layer. The neuron’s operation consists in calculating the following equation

$$ f\left( \sum\limits_{i}w_{i}x_{i}\right), $$
(3)

where wi is the weight set on the connection between the current neuron and i-th in previous layer, and xi is incoming value from i-th neuron. The sum of products is recalculated by activation function f(⋅). The number of columns with such neurons depends on given problem and the number of output information. We assume that the number of output neurons is determined by the number of features in classification task. In the network thus created, the values of weights on the connections between the neurons are generated in a random manner from the interval 〈0,1〉.

The training process of the classifier consists of modifying weights to minimize error on the network output. We use cost function to optimize weight in algorithm called RMSProp which is based on calculation of two statistical coefficients in each iteration t. The first of them is variation defined as

$$ m_{t}=\beta_{1}\cdot m_{t-1}+(1-\beta_{1})\cdot \nabla f(w_{t-1}), $$
(4)

where β1 and β2 are parameters. The second one is the mean defined as

$$ v_{t}=\beta_{2}\cdot v_{t-1}+(1-\beta_{2})\cdot \displaystyle\left( \nabla f(w_{t-1})\right)^{2}. $$
(5)

Both measurements are used in changing the value of weights as

$$ w_{t}=w_{t-1}-\displaystyle\frac{\gamma}{\sqrt{v_{t}+\epsilon}}\odot m_{t}, $$
(6)

where γ is constant value understood as training rate, 𝜖 ≈ 0 prevents division by zero.

4 System architecture

Proposed soft trees allow to create classification system for excavation found objects. We assume that the user works with smartphone built-in camera. In addition, we assume that the photo is taken by simple a application which performs calculations and returns the result on the screen. Classified photo is added (after obtaining the user’s consent) to external database where it extends general knowledge of the system. Using this general knowledge, all the classifiers in nodes of the soft tree are re-trained on what makes them increasingly sophisticated for excavation artifacts. This way of constant knowledge upgrade allows updating the weights and structure of the soft tree on local user devices using only a little data transfer for much more accurate classification. Accuracy is higher because training involves more new information obtained from users in various conditions so the general knowledge of the system is growing fast. In Fig. 5, the data exchange between users and an external server responsible for storing new objects and re-training the classifiers is presented.

Fig. 5
figure 5

Data flow in the proposed system architecture. When the user takes photo by his/her camera, the object is classified by the use of soft tree, which base decision in each class on the result of trained CNNs. This classification result is returned to user as recommendation on the excavated object. Parallel to this proposed system, it takes the new photo and revises the knowledge during training

Full size image

4.1 Dataset

In our experiments, we used a general data-set composed of 750 images of different important museum exhibits (artifacts) and objects from these places but not marked as important artifacts. We have used them to train our system to differ artifacts from regular objects. All data images were described using different specified features summarized in the following form

  • dated age: primeval, antiquity, middle age, modernity

  • material: metal, stone, wood, clay, other

  • surface condition: smooth, rough, other

  • type: tool, pottery, weapon, other

Sample images are presented in Fig. 2.

4.2 Soft tree composition

Soft tree was constructed using defined in above subsection categories Fig. 3. As first node dated age was chosen. This is an important start point since in most cases, the age of artifact defines many other features which are typical for various historical periods. Therefore, we are able to minimize the number of possible cases in soft set binary table.

Each of the features in decision classes was given a priority weight, which represents importance of each feature for decision in each class. Priorities are set to each feature and easy to recalculate in entire system since they do not need any special models but an expert in the field of archeology to set them in each experiment. The importance of evaluated artifact is measured in accordance to

$$ \sum\limits_{j=1}^{J}\frac{\max\limits_{i\in 1 {\dots} I}w_{i,j}\cdot h_{i,j}+\min\limits_{i\in 1 {\dots} I}w_{i,j}\cdot h_{i,j}}{2}, \text{ where } h_{i,j} \neq 0, $$
(7)

which means that we are calculating the sum of weighted decisions from soft tree nodes hi,j where \(i\in 1 {\dots } I\) by selecting only those for which weighted importance wi,j in decision class \(j\in 1 {\dots } J\) is highest and lowest. Therefore, summary evaluation of artifact represents averaged evaluation of important features. If we have exact classification only for highest importance features, the artifact is rated high while if some classification is also possible for low importance features, also the summary evaluation of the artifact is lower.

For example, when classification process returns modernity age, we can rather assume lower importance of this artifact. Similarly when dated age is primeval, it gives high importance, no matter the type of the artifact since from this period everything is important excavation artifact.

4.3 Applied convolutional neural network

Proposed architecture of the neural classifier is presented in Table 1. For each feature evaluation, we have used the same architecture; however, different filtering was applied. We have used two different kinds of filters in convolution layers. For decision over age and type of the excavation object, we have applied sharpening in first CNN convolution layer and edge detection in second one, while for decision over material and surface condition we have applied double sharpening (Fig. 6). Figure 7 presents how described filters change the initial image. We have decided on this filtering combinations since to evaluate various features from excavated objects, we need to concentrate CNN on different aspects. The general features of objects are visible from shape so we have used a combination of filtering which emphasize them, while to decide on material only, the structure of the surface is necessary. In both cases, we received good results. Applied pooling layer uses two different dimensions of the window. First we use 3 × 3 Max-pooling window and in second layer 2 × 2 Max-pooling window. These sizes help to maintain the most important features of excavated objects for final verification in fully connected layers. The whole idea of CNN processing is presented in Fig. 6

Table 1 Applied convolutional neural network architecture for n outputs and 3 × 3 filtering windows and two different pooling window sizes
Full size table
Fig. 6
figure 6

Applied construction of convolutional neural network. Depending on decision for the soft tree, we use two models of filtering. For decision in soft tree categories of age and type of the excavation object, the filtering is based on sharpening kernel and is followed by edge detection kernel from Sobel-Feldman. For decision in soft tree categories of material and surface condition, we use double sharpening kernel. For both models of filtering, we have applied 3 × 3 window

Full size image
Fig. 7
figure 7

Sample presentation of two different kinds of convolution filters. For decision over age and type of the excavation object, we use sharpening in first CNN convolution layer and edge detection in second one, while for decision over material and surface condition, we use double sharpening

Full size image

5 Research results

In this section, we present results from sample implementing and testing of our proposal. Each soft tree node was composed of one convolutional neural network for classification. Each of these networks was of the same construction (see Table 1) and was trained with the same data through 100 iterations. The difference was in the labels for each of decision classes depending on expected classification of a given feature. If sample image completed classification, the obtained data (expressed as decision for the best fit of features) was placed in a 16-element vector (if classified given one otherwise filled with zero). Subsequently, the inference soft table was made. For presentation of research experiment purposes, we show just a fraction of the table, which can be seen in Table 2. It is worth noting that in the case of calculating all possible decision cases, the binary table of the soft set has 216 = 65536 cases to be evaluated. For sample discussion, we have selected results of classification for the artifacts from different times ranging from Ancient through Medieval finishing in our recent history. We have trained proposed CNN architectures using classic fold cross validation model in which information used to training is different in each class and after each training iteration, we change classes for better classification abilities of the CNN architecture. Figure 8 presents sample results of CNN classification in decision classes on each level of soft tree. In each case for decision on the input data, we have used maximization function described in (7). The results of evaluation were compared with other results in the table to make decision. Table 2 presents sample work of our proposal.

Table 2 A sample record of binary table for proposed soft set decision classes used in evaluation of soft tree decision
Full size table
Fig. 8
figure 8

Sample excavation artifacts recognition processes. Proposed soft tree decision classes are related to each level in the soft tree. Decision in-between the class depends on result from CNN recognition, which is processing the image to evaluate it in categories of the decision class. On each level of the soft tree, different CNNs work to recognize different features. Final decision of the classifier depends on composition of all results from all levels compared with rules in knowledge base using assumptions of soft sets

Full size image

In Fig. 9, we can see classification results on each level of the soft tree, where trained CNNs were used to classify the object from the input image between categories of decision class. In each decision class, some of the samples were misclassified. The reasons for these mistakes of CNN were in similarities of objects features like materials or shapes, which reminds some similarities of others. However in general proposed architecture achieved good classification results. Let us discuss sample results of CNN values in each level of soft tree (decision class) for samples presented in Fig. 8. We can see that both Ancient Greek sculptures were classified to artifacts dated in primeval or ancient times. Stone or clay were proposed as material, while surface condition was not so easy to evaluate. Indeed from both images, we can see that some of regions are not in the best condition, which is reasonable for ancient artifacts. CNN classifiers placed them in no special group of use. Medieval sculpture was fitted to various periods, which is true since also now wooden sculptures are very often produced. Material was suggested as wood or other which results from wood but also paint of the sculptures. Structure of the artifact is rough presenting many details. CNN also suggested that this artifact is not weapon neither any of housewares. Ancient pottery was dated in oldest periods what results from material of this construction. CNN recognized it as stone since there are many small stones in cement walls of this pottery. Walls are rough due to stone elements that was recognized by the classifier. Category for this artifact was among housewares like tools or pottery. Let us now see how proposed CNN classifiers worked with two modern elements. A crocodile scull was however dated to primeval or antiquity which we can encounter as failure, since it is not older than 60–80 years. This can result from the shape of this scull which reminds a warrior mask from ancient times. Material was classified as other, which is correct since color and structure are different from clay and stone. Due to shape, also condition was proposed as rough. The category was presented as weapon or other, these choices correlate with previous propositions. Another modern artifact was metal element from II World War. Here proposed CNN correctly defined its age as modern. Similarly material was defined as metal that results from color and structure of this object. CNN proposed weapon or tool as category. It can be true since in our collection are many metal tools or weapons so the classifier based decision on general knowledge from data. Values of these classifications were composed into vectors for soft set. In Table 2, we can see how these values describe our sample artifacts, while the rest of the elements represent a fraction taken from general knowledge table to show how the reasoning in our proposal is done.

Fig. 9
figure 9

Sample excavation artifacts recognition processes. Proposed soft tree decision classes are related to each level in the soft tree. Decision in-between the class depends on result from CNN recognition, which is processing the image to evaluate it in categories of the decision class. On each level of the soft tree, different CNNs work to recognize different features. Final decision of the classifier depends on composition of all results from all levels compared with rules in knowledge base using assumptions of soft sets

Full size image

For reasoning from CNN classifications, we have calculated soft table using priority wights. Presented numbers of wi,j were assigned according to our subjective opinion, but because of the open soft tree structure, they can be changed at any time according to an expert in the field of archeology. We use some numerical values here just to show how proposed reasoning works. To do the reasoning, first we calculate score number of each object using proposed (7). In Table 2, objects from general base were sorted in accordance to these values. This helped us to create a range of affinity for important objects (artifacts) and not important ones (not artifacts). From presented values, we see that minimum score for evaluated object to be recognized as artifact is 3. All objects under this score were just some objects from historical places but not so important for excavation or historical value. If we compare calculations for images of objects from Fig. 8, we can see that proposed soft tree reasoning gives positive results for archeological artifacts from Ancient and Medieval periods, while objects like crocodile scull or metal elements were evaluated under score level so we assume these were not classified as artifacts.

6 Conclusions and future works

Proposed novel decision support is based on our idea of soft tree composition. This method is simplified reasoning form, where we use decision classes represented in consecutive levels of the soft tree. Each level can be composed of number of nodes, where each of them represents evaluation of one feature in decision class. Due to proposed reasoning, we can freely add/remove nodes (features in decision class) or even entire levels (entire decision classes), since these operations do not influence our reasoning method. Proposed reasoning needs correct evaluation of features in each node. In our case for images, we have used trained convolutional networks. This element of the proposed decision support is developed for each feature separately, and is trained using general knowledge, which means that adding new nodes results in composing and training new CNN. Decision-making process is based on calculating scores for each of the elements in general knowledge base and compare with these scores new evaluated objects. This process is very simple and we have proposed a new simple equation to calculate the score by using priority weights for each of features. Proposed weights can be changed in any time in accordance to new expert knowledge or conditions of experiment. Reasoning from the soft set table is based on these values, where we simply compare scores of new elements to decide.

Results show that proposed method is very simple to use, and also efficient. Images of Ancient or Medieval artifacts were correctly classified as important, while modern objects from this century were given lower score placing them among regular objects.

Our novel proposed method works well for decision processes over cultural heritage. Just by using a smartphone camera, we are able to evaluate objects for their cultural value. Reasoning is very simple, and the process is flexible to any changes in features which we want to present for evaluation. On the other hand, some changes and improvements would be beneficial. Future research will concentrate on changes to the final decision rule, so it would be defined in more flexible way to introduce more options for final decision. In proposed form, we see it works as very simple yes/no rule, if the score is above or under decision level. From our initial research, we think that fuzzy reasoning on this decision level value would be efficient. Fuzzy membership function would enable us to adjust flexibly decision and compose sophisticated rules, which would simulate human thinking.