Abstract

Along with the rapid development of remote sensing satellites and sensor network technology, vast amounts of remote sensing imagery and in situ observations have been accumulated. Further, various researchers and agencies have released a variety of thematic image products. These heterogeneous observations are therefore difficult to utilize comprehensively. In this study, an ontology-based framework for integrating remote sensing imagery, image products, and in situ observations was developed. It was extended based on the Semantic Sensor Network (SSN) ontology in the Web Ontology Language (OWL). The detailed process of ontology construction and rule establishment was demonstrated. Combined with some actual remote sensing imagery, image products, and in situ observations, semantic queries based on DL Query and SPARQL were conducted to establish the rationality and feasibility of the ontology and framework.

1. Introduction

With the development of remote sensing technology and the improvement of the resolution of satellite sensors, the application of remote sensing to quantitatively obtain the required parameters on a large scale on the ground has become even more extensive. However, remote sensing, as a monitoring method for periodic acquisition, lacks timeliness and is not suitable for monitoring hot spot regions with rapid variations, and because of atmospheric interference, the overall accuracy is not sufficiently high. As a kind of ground-based monitoring system, wireless sensor networks and remote sensing data functionally complement each other and can dynamically monitor all kinds of parameters needed by the region in quasi-real time. Wireless sensor network technology is not only regarded as a new outgrowth of traditional remote sensing technology but also an extension of remote sensing, the geographic information system, and the global positioning system [1, 2]. Therefore, the integration of the technical characteristics of both, each playing to their respective advantages, will become a development and application trend in the future [3, 4].

A large number of studies have focused on integrating remote sensing and ground-based sensor networks [510]. A collaborative inversion method combining a ground sensor network and HJ-1 satellite remote sensing data was proposed to retrieve ground temperature [3]. In situ temperature measurement and thermal imaging from flyovers or drones or satellite remote imaging were used to track the dynamics of freshwater–seawater mixing behavior in a bay [9]. Warming trends of perialpine lakes were analyzed based on homogenized time series of historical satellite and in situ data [8]. A novel strategy combining global satellite multispectral datasets, environmental constraints, and in situ acquisition of geospatial data was presented for cash-crop mapping [7]. From the contents of these studies, it can be seen that research on integrating remote sensing images and ground sensor networks is often aimed at a single target, using specific remote sensing images. It is difficult to carry out composite and integrated research.

Constructing uniform resource representations is quite significant to improve resource sharing and interoperability, as well as the comprehensive process of decision-making [11]. Semantic Sensor Network (SSN) ontology [12], proposed by the World Wide Web Consortium (W3C), has been widely used as an ontology model to solve semantic barriers in sensor networks [1321]. However, SSN ontology lacks descriptions about remote sensing imagery, and satellites only act as platforms for sensors or actuators. There have been several studies on the semantics of satellite-based sensors or remote sensing imagery. An ontology for associating sensor observation capability was developed to facilitate remote sensing satellite selection in certain earth observation tasks [22]. A light ontology was constructed to represent the content of remote sensing imagery based on existing geographic ontologies, the domain-related corpus, and the WordNet [23]. The WordNet, identifying concepts having a similar meaning but using different words, has been widely applied in evaluating the semantic similarity between words [2426]. A two-phase methodology was built to facilitate the satellite parameterized description and remote sensing data classification based on the ontology concept [27]. Taking into the principal scaling factor and various spectral variability consideration, a novel spectral mixture model was proposed to improve the flexibility of the endmember dictionary by introducing the dictionary of spectral variables [28]. Spectral rules have also been used for semantic classification of remote sensing imagery [2931]. In order to make better use of satellite image resources, the World Meteorological Organization (WMO) has provided most mission objectives for major satellite sensors [32]. However, semantic research about the combination of remote sensing images and ground sensor networks is still rare.

At present, a large number of image products based on remote sensing imagery have emerged. These image products are based on various remote sensing images, using object identification, parameter extraction, and other methods in combination with field measurements or referring to existing datasets if possible. After band selection and fusion, image enhancement processing, stitching, or cutting, these image products represent certain thematic information, such as land use/land cover, precipitation, and surface water. Although image products and remote sensing imagery are similar in structure, the specific means of utilizing them are completely different.

Considering the abovementioned themes, the objective of this study was to develop an ontology with semantic rules to integrate remote sensing imagery, image products, and in situ observations (named RSISO ontology) based on the W3C OWL 2 Web Ontology Language [33] and the software Protégé developed by the Stanford Center for Biomedical Informatics Research at the Stanford University School of Medicine [34]. The ontology was extended from the SSN ontology based on the concepts of spatial pixel and spectral rule. This new ontology can be utilized to retrieve available and potentially valuable remote sensing imageries, image products, and in situ observations. Specifically, the contributions were summarized as follows: (1) an ontology-based framework was proposed to integrate remote sensing imagery, image products, and in situ observations; (2) the formal expression of the characteristic index is more conducive to the sharing and reuse of empirical knowledge in different fields; and (3) available data can be discovered more easily by using semantic technology. A new perspective will be provided by applications of this ontology, making sufficient use of remote sensing imageries, image products, and in situ observations. It will be powerful in serving complex and comprehensive environmental monitoring, disaster management, and decision-making processes.

The remainder of this paper is organized as follows. Section 2 analyzed the typical remote sensing images and image products; then, the ontology and related rules were constructed. The experimental data and results were described in Section 3, the discussion was presented in Section 4, and conclusions and future work are given in Section 5.

2. Materials and Methods

2.1. Overview

The RSISO ontology was designed for integrating remote sensing imagery, image products, and in situ observations. Typical instances of these three types of observation data are listed in Table 1. There are three dimensions of semantics in blending the three kinds of observation data: spatial, temporal, and thematic. It acts as a semantic framework to achieve validation, fusion, and provenance among the three kinds of observation data, as illustrated in Figure 1. The detailed steps of constructing the ontology are presented in the following subsections.

2.2. Procedure of Ontology Modeling

It is necessary to establish an ontology in a standard process in order to ensure the rationality and reusability of the constructed ontology. The seven-step methodology proposed by Noy and McGuinness of Stanford University [37] was applied in this study. The seven-step methodology has been widely used [3843] and is closely integrated with the ontology construction software Protégé [34]. Therefore, it was also used in modeling the RSISO ontology. The details are as follows.

2.2.1. Determine the Domain of the Ontology

The formal representation of earth observation was determined as a domain with scopes limited to remote sensing imagery, image products, and in situ observations, which were examined in Table 1 and Section 2.1.

2.2.2. Consider Reusing Existing Ontologies

Because the SSN has had many successful applications in ground-based sensor networks, as described in Section 1, we reused the core concepts of SSN. Additionally, GeoSPARQL [44], Time [45], and WordNet [46, 47] ontology were imported as semantic descriptions of space and time, respectively. For ontologies from different sources that were reused, it is necessary to provide a clear list of prefixes, as shown in Table 2.

2.2.3. List Important Terms in the Ontology

The ontology’s key terms for the application domain were defined and used for the class definition in the next step. These terms include remote sensing imagery, image product, spatial pixel, band, and characteristic index. Descriptions, key properties, and examples of these terms are listed in Table 3.

2.2.4. Define the Classes and Their Relations

Based on terms enumerated in the former step, five classes—RemoteSensingImage, ImageProduct, SpatialPixel, Band, and CharacteristicIndex—were created in the proposed ontology. RemoteSensingImage and ImageProduct were set as subclasses of sosa:Result, SpatialPixel was a subclass of sosa:FeatureOfInterest, and Band was a subclass of sosa:ObservableProperty. CharacteristicIndex, as a property, cannot be observed directly, so it was inherited from ssn:Property.

2.2.5. Define the Object Properties of Classes

Object properties of classes are used in OWL 2 to define relationships between different classes of individuals. Because the classes created in the RSISO ontology are subclasses of the classes existing in the SSN ontology, their related object properties were inherited from the SSN ontology. The class hierarchy and their relationships are illustrated in Figure 2.

2.2.6. Define the Data Properties of Classes

Data properties of classes are used in OWL 2 to connect individuals with literals. In this step, we defined value type restrictions of data. For example, the value of property hasID is restricted to be of datatype string and datatype double for the value of property hasCloudCover.

2.2.7. Create Instances

Individual instances of classes in the hierarchy were created in the last step. In the RSISO ontology, instances mainly refer to kinds of actual remote sensing images, image products, and observation data from the in situ sensor. The parts of instances and their important properties are listed below.

2.3. Rules in the RSISO Ontology
2.3.1. Spatial Unification

Since different coordinate systems are used in remote sensing imagery, image products, and in situ observations, the first step is to unify the Worldwide Reference System (WRS), the Military Grid Reference System (MGRS), and other coordinate systems into WGS 84. There are tools and codes to achieve such coordinate conversion, such as the United States Geological Survey (USGS)’s Landsat Acquisition Tool [57] and converting coordinates with Java code provided by Salkosuo [58]. As such, a detailed conversion procedure is not described here.

As described in step 2 in Section 2.2, GeoSPARQL was used to implement spatial semantic queries in the RSISO ontology proposed in this paper. rsiso:RemoteSensedImagery and rsiso:ImageProduct were defined as a subclass of sosa:Result in Section 2.2, and they rsiso:hasPixel rsiso:SpatialPixel. Then, rsiso:SpatialPixel and sosa:FeatureOfInterest were set as rdfs:subClassOf geo:SpatialObject. Next, the topological relations between spatial objects, such as intersects, within, or contains, were established by the Topology Vocabulary Extension (relation_family) parameterized requirements class of the GeoSPARQL. The spatial semantic extension of RSISO through the GeoSPARQL is shown in Figure 3.

2.3.2. Temporal Rule

In the time dimension, in situ data have a higher resolution, such as hourly precipitation data that represent rainfall within an hour of the day, while for remote sensing image data, based on the return period of the carrying satellite platform, there are different temporal resolutions, such as 10 days of Sentinel 2 MSI, where the imagery typically represents the result of the day and the temporal resolution of the image product varies (e.g., JRC Monthly Water History in months, JRC Yearly Water Classification History in years, and CHIRPS Daily). Time ontology can be used to define these times (instants or intervals). As shown in Figure 2, these remote sensing data and remote sensing products obtained the range time:Temporal Entity of the sosa:phenomenonTime object property or xsd:dateTime value by inheriting the sosa:Observation class. They were achieved in the W3C SSN ontology.

2.3.3. Thematic Rule

Regarding the theme, in situ observation has explicit content information. For instance, rain gauges observe an amount of precipitation. Similarly, the application fields of the image products can be easily found because they are produced for specific topics, such as JRC Monthly Water History for surface water and CHIRPS Daily for precipitation.

Although the WMO has provided most of the application topics for remote sensing satellites, it does not explain the internal relationship between remote sensing satellites and application topics, so it is unable to expand the application field of remote sensing satellites according to a wide range of practical applications. Spectral rules were used for the classification of remote sensing images, and effective results were obtained [7, 29, 30]. Therefore, spectral rules were introduced into the RSISO ontology through the class RSISO:CharacteristicIndex.

According to the bands contained in the specific remote sensing images, the available characteristic indexes can be inferred. For example, according to the calculation formula of the Normalized Difference Water Index (NDWI), the green and near-infrared bands were needed to acquire the index. From the bands listed in Table 4, both Landsat 8 OLI and Sentinel-2 MSI can obtain the NDWI, and sequentially, the land cover of surface water can be obtained. The difference is that the near-infrared band corresponds to B5 and B8 in Landsat 8 OLI and Sentinel-2 MSI, respectively.

Considering the text description difference and semantic correlation of application objectives, the WordNet ontology [59] was introduced in the RSISO. The WordNet ontology, a lexical database of the English language, enables acquiring synonyms or related words through semantic relevance [60]. More available data can be discovered by extending the requirement text and matching the instances of sosa:Observation of in situ observation, RSISO:Band of image product, and RSISO:CharacteristicIndex of remote sensing imagery. The structure of the thematic matching in RSISO is shown in Figure 4.

3. Results

3.1. Implementation of the RSISO Ontology

The RSISO ontology designed in Section 2 was created with Protégé (version 5.2.0). The querying sentences were written in the SPARQL Protocol and RDF Query Language (SPARQL) [44] and executed by the Protégé SPARQL Plugin (version 2.0.2). HermiT (version 1.3.8.413) was selected as the reasoner, and the rules for reasoning were encoded in Semantic Web Rule Language (SWRL) [61] and implemented by SWRL Tab Protégé 5.0 + Plugin (version 2.0.4) in Protégé.

As shown in Figure 5, the ontology determines the classes and their relationships, which define different types of observation data and information available from these results. The classes ImageProduct and RemoteSensedImagery were designed as a subClassof class Result of SSN. They have the hasPixel property to describe the SpatialPixel contained in them. The difference between them is that the Band contained in ImageProduct refers to specific application objectives; however, Band in RemoteSensedImagery represents the reflection value in a particular band range. For example, the values of the “water” band in the ImageProduct JRC Monthly Water History stand for permanent water, seasonal water, or no water. In contrast, the values of Band in RemoteSensedImagery cannot be used directly to identify the type of land surface cover. The values of particular bands can be used to obtain the land surface cover index, such as the NDVI and NDWI, through a mature and reliable normalized calculation method. In the proposed ontology, CharacteristicIndex represents the type of surface cover that can be inferred from the bands that RemoteSensedImagery has. Because CharacteristicIndex cannot be observed by sensors, it was designed as a subClassOf Property. The parts of classes and properties created by Protégé are shown in Figure 5.

Classes or individuals are linked through the object properties defined in the ontology. These object properties, as interactions between different objects, are the significant elements in the ontology. The main object properties in the proposed ontology are listed in Table 5.

3.2. Inference Based on Rules

Rules were defined in Protégé to provide principles for logical reasoning based on experiential knowledge. Experiential knowledge originates from the long-term practice of professionals. The ontology was proposed to find the potential application domains of remote sensing imagery, image products, and in situ sensors. The in situ sensors themselves have clear observation targets, and the image products can also distinguish their meaning by the band names. Therefore, the rules are mainly used to discover the potential application domains of remote sensing images by calculating different land cover indices. The parts of the rules created for the ontology are listed below.

Rule 1. Based on NDVI, if an instance of RemoteSensedImagery contains observedProperty Band_NearInfraredRed_LandSurface and Band_Red_LandSurface, it can be inferred that the instance hasCharacteristicIndex Index_NDVI. The rule was written as (1). where ?o means the observation instance we want to obtain, Band_NearInfraredRed_LandSurface and Band_Red_LandSurface represent two instances of class Band, and Index_NDVI means the Normalized Difference Vegetation Index, an instance of the class CharacteristicIndex.

Rule 2. Based on the NDWI, if an instance of RemoteSensedImagery contains observedProperty Band_Green_LandSurface and Band_NearInfraredRed_LandSurface, it can be inferred that the instance hasCharacteristicIndex Index_NDWI. The rule was written as (2). where ?o means the observation instance we want to obtain, Band_Green_LandSurface and Band_NearInfraredRed_LandSurface represent two instances of class Band, and Index_NDWI means the Normalized Difference Water Index, an instance of the class CharacteristicIndex.

Rule 3. Based on the EVI, if an instance of RemoteSensedImagery contains observedProperty Band_NearInfraredRed_LandSurface, Band_Red_LandSurface and Band_Green_LandSurface, it can be inferred that the instance hasCharacteristicIndex Index_EVI. The rule was written as (3).

We can input “Observation and hasCharacteristicIndex value Index_NDVI,” “Observation and hasCharacteristicIndex value Index_NDWI,” or “Observation and hasCharacteristicIndex value Index_EVI” in the DL Query tab of Protégé to acquire all imagery instances which can be used to obtain computable NDVI, NDWI, or EVI values.

Similarly, other rules can be built on the basis of existing experience and knowledge.

3.3. Semantic Query Using the RSISO Ontology

In order to evaluate the RSISO ontology, a series of semantic querying and reasoning was designed. The querying language was encoded in the SPARQL and executed in the SPARQL Query tab of Protégé.

On the one hand, we can acquire various heterogeneous observations which are observed in a specified period. For example, all observations, including between “2015-08-23” and “2015-09-10,” can be obtained by following the SPARQL sentence based on the presented RSISO ontology model. The results are shown in Figure 6.

PREFIX xsd: <http://www.w3.org/2001/XMLSchema

PREFIX :http://localhost/RSISO1010.owl

SELECT ?observation ?datetime

WHERE

{

?observation :resultTime ?datetime .

FILTER (?datetime >= "2015-08-23T00:00:00+08:00"^^xsd:dateTime && ?datetime <= "2015-09-10T00:00:00+08:00"^^xsd:dateTime)

}

On the other hand, sosa:observedProperty can be used to obtain the properties of in situ observations, such as water level and flow volume and precipitation. Combined with the rule definition for remote sensing imagery in Section 3.2, observations with similar or complementary properties can be found, such as the water field in JRC Monthly Water History and NDWI in Landsat 8 OLI imagery. In addition, the retrieval of heterogeneous data can be achieved according to spatial conditions, by introducing the functions of GeoSPARQL. Combining the semantic query conditions of time, space, and topic, the specific available data can be retrieved. Combining the formulation of reasoning rules and the semantic conditions of time, space, and topic, the specific available data can be obtained more easily. This is beneficial for the fusion analysis of satellite–earth observations.

4. Discussion

4.1. Integrated Application with In Situ and Remote Sensing Observation and Image Products

There are an increasing number of in situ and satellite-based sensors, and a large number of image products using remote sensing imagery are emerging. This provides a great opportunity for research work, such as satellite–ground cooperative monitoring, image product correction, and remote sensing image inversion evaluation. At the same time, it also brings challenges to the management mode, association technologies, and query methods of the heterogeneous observed data. Up to now, some studies have applied spectral rules to the automatic application of remote sensing images [29, 30] without, however, considering in situ observations. The proposed ontology model, based on the mature SSN ontology for managing in situ sensor networks, extends the objects of remote sensing imagery and image products so that these observation results can be uniformly managed and queried, which will be very beneficial to the integrated application of heterogeneous observation data.

4.2. Prior Knowledge Expressed in SWRL Can Be Extended and Shared Better

The reasoning capability provided by RDF description logic and SPARQL is limited. On the basis of OWL, we established some rules based on SWRL, so as to improve and optimize knowledge reasoning. Furthermore, based on the formal expression of SWRL, which is a W3C specification, experts in different fields, such as precision agriculture, climate change, extreme weather, and emergency decision-making, can transform their knowledge into the rules of ontology so that reusing and sharing their knowledge becomes feasible.

4.3. Image Products Can Be Used More Widely Based on Semantic Queries

With the accumulation of remote sensing image data, different kinds of thematic image products have been released by scientists or institutes, for example, Global Surface Water by the European Commission’s Joint Research Centre [62] and Rainfall Estimates from Rain Gauge and Satellite Observations (CHIRPS) by the Climate Hazards Center at the University of California, Santa Barbara [63]. These image products have great reference value for related research, but because they are professional and not popular, their role is not easy to be recognized. The ontology proposed in this paper provides a channel to manage and discover these imaging products, taking into account both in situ and remote sensing observations.

5. Conclusions

In this work, we developed an ontology-based framework for integrating remote sensing imagery, image products, and in situ observations. It was developed by expanding the W3C SSN ontology with spatial, temporal, and thematic rules. The detailed process of ontology construction and rule establishment was demonstrated. Combined with some actual remote sensing images, image products, and in situ observations, some semantic queries based on DL Query and SPARQL were implemented, and the rationality and feasibility of the ontology and framework proposed in this paper were proved. In addition, more rules can be built up by SWRL based on existing experience and knowledge. It will be beneficial for the reuse and sharing of professional knowledge and taking full advantage of impact products.

The current work realizes the ontology prototype of integrated management and preliminary reasoning of remote sensing images, image products, and in situ observation data. However, more semantic obstacles will be encountered in the comprehensive application of multisource data. The ontology proposed in this paper needs to be further integrated with the existing ontology and improved in more complex engineering projects.

Data Availability

The ontology file developed in this study is available from the first author (Chao Wang, [email protected]) upon request.

Conflicts of Interest

The authors declare no conflict of interest.

Acknowledgments

This work was supported by grants from the National Key R&D Program of China (No. 2018YFB2100603), the National Natural Science Foundation of China (NSFC) Program (Grant Nos. 41601406 and 41890822), and the Fundamental Research Funds for the Central Universities (No. 2042019KF0211). This work was conducted using the Protégé resource, which is supported by grant GM10331601 from the National Institute of General Medical Sciences of the United States National Institutes of Health.