1 Introduction

Smart coal mining is the key to solving problems associated with the continuous complexity of coal mining and geological conditions (CSIRO 2019; Semykina et al. 2017). Next-generation information technologies, such as fifth-generation (5G), artificial intelligence (AI), big data, cloud computing, and virtual reality (VR) technologies (Wang et al. 2019) need to be integrated into the construction process of smart coal mines (Jiang et al. 2020; Kopacz et al. 2020). Smart mining technology is the process of recovery operations that are performed independently by mining equipment without direct human intervention through the smart perception of a mining environment, smart regulation of mining equipment, and autonomous mining operations. Developing coal mine robots is crucial to solving the card neck problem (Ge et al. 2020a, b, c; Ralston et al. 2015). Major coal-producing countries, such as China (Wang et al. 2017), Australia (Ralston et al. 2017), and Poland (Boloz et al. 2020), consider robotics research as imperative, which aims to comprehensively improve the level of robotization of underground equipment and provide enhancements for smart mining.

In Poland, Jonek-Kowalska (2019) established a model to improve the efficiency and productivity of coal mining companies and highlighted the future direction of mining robotization. They also developed a robot mobile detection platform and a separate mobile platform for inspection, which could detect hazardous areas of methane and coal dust explosions in hard coal mines (Kasprzyczak et al. 2013, 2016). Noort et al. (2008) highlighted that robotic operations for coal mining are crucial for automated underground and unmanned mining. Novák et al. (2018) and Ray et al. (2016) analyzed the specific design of mobile robots for underground reconnaissance. Doroftei and Baudoin (2012) proposed the concept of a walking robot for mining, Ranjan et al. (2019) developed a wireless network to secure search and rescue robots in underground mines, and Bołoz et al. (2013, 2018) conducted an exploratory study on the automation and future robotization of shearers.

In China, Ge et al. (2020a, b, c) proposed the broad concept of coal-mining robots and presented the development status and critical issues. Traditionally, the focus of coal-mining robots has been specific scenes, such as disaster relief, search and rescue, and critical geological exploration issues (Tan et al. 2017; Ge et al. 2017; Zhao et al. 2017). Recently, digital perception technology, including AI, scanning, and detection technologies, has been integrated into the industry; thus, mining robots will gradually have advanced AI perception capabilities (Yang et al. 2020). To solve the problem of autonomous navigation for underground mobile robots, Ma et al. (2020) proposed a visual reconstruction approach. Song et al. (2020) introduced bionics to achieve efficient walking, intelligent perception, and control. Li et al. (2020) designed an inspection robot for face mining. Chen et al. (2020) studied underground robot operating environments and navigation environment map construction. Huang et al. (2017) proposed the architecture of an integrated intelligent control platform for Ethernet/IP-based general mining robots. These works have made significant progress in developing mining robots from key technologies in robot designs to complete machine testing.

The production system for a fully mechanized coal mining working face is crucial for coal production. Its characteristics are as follows: (1) There are many types of equipment, and their cooperative operational relationships are complex. The degree and basis of intelligence for each piece of equipment also differ, and the underground environment is usually unknown. (2) Current research focuses primarily on multi-point single equipment and local technologies. The equipment involves sensing, decision-making, control, and other issues. In addition, the research and development (R&D) cycle is lengthy with a high requirement for underground explosion-proof performance, which makes collaborative research and system integration more difficult. Thus, planning and designs need to be performed from the top to accelerate R&D efficiency.

Unified, smart, fully mechanized mining test and evaluation software is required to grasp the global and local operational state of a smart cooperative large-scale system (Wang et al. 2020a, b; Zhang et al. 2017). Brzychczy et al. (2014) designed software to support the design of certain elements of the coal mining process, which included two fuzzy systems: fire safety evaluation system (FSES) and functional safety over EtherCAT (FSOE). The FSES performs equipment selection to support long-walled working faces, whereas FSOE supports estimations of the production results. The equipment selection results are obtained based on the parameters of the working face and a knowledge-based system for longwall mining. The obtained values are then inserted into the FSES for planning and evaluation with promising results. Employing the concept of robotics and expert systems, Brzychczy et al. (2011, 2017) established an underground, hard, coal mine planning and optimization system.

The main problems and solutions from the current development perspective are as follows:

  1. (1)

    The software and actual working conditions need to be closely integrated. The current test software is not based on real data, and the simulations lack credibility. Thus, the results are far from reality, and the visualization effects are poor. The historic data of mining working faces should be exploited to construct virtual scenes with a high simulation degree.

  2. (2)

    Comprehensive simulation test software needs to be constructed as the current software functionality is relatively singular. The selection and design of working faces, equipment operational status, mining technologies and methods, and coal seam geological models need to be analyzed using specialized software. Tools for unified visual analysis and measurements are inadequate, and the constructed system should integrate these functions into software. A scientific evaluation system should be established, and unified measurements and evaluations should be performed.

  3. (3)

    Advanced virtual simulation methods need to be employed. Currently, there are no complete user input systems, and the constructed virtual simulation equipment and scenes are unreliable. The problems of perception, decision-making, and control in actual equipment operations suggest that most typical problems are from the insufficient measurement accuracy of sensors, delays in network transmission systems, error operations of single programmed decision-making systems, lack of action from control elements, etc. The virtual simulations would consider these factors for high-quality simulations by exploring digital twin (DT) technology and related concepts. The relevant virtual mining designs are performed before the design or operations of a working face. The overall operation is first simulated before use as a reverse drive to guide the design.

Currently, DT technology has been integrated into the field of coal mining (Xie et al. 2019a) and has become a major technical aspect of smart mining (Ge et al. 2020a, b, c; Xie et al. 2019b). A DT model can be built to truly reflect the operations of robot equipment for a working face (Wang et al. 2020a, b; Xie et al. 2019c). The software can provide technical support for the design, manufacture, and R&D of mining robots (Xie et al. 2019d). To address these issues, this study proposes a collaborative operation of a virtual general mining robot production system based on AI, DT, and VR technologies. This could replicate the virtual mining operations at a specific working face and create fully mechanized virtual mining production with self-organizing and cooperative operations based on actual operational data of geology and equipment in the working face (Xie et al. 2019e; Shi et al. 2020). Different parameter inputs and smart operational modes allow simulating and evaluating the current and future operating conditions of working faces. We study the overall architecture, working historical data processing, offline-driven virtual operations, operational system, and mining evaluation methods. A complete prototype system is designed in three-dimensional (3D) views including virtual, physical, and virtual–real mapping.

2 Overall framework

A fully mechanized Unity3D simulator (FMUnitySim) is developed, and some multiagent-based models for the mining equipment and environments are established. The FMUnitySim can simulate fully mechanized coal mining equipment operations and overall processes under various input conditions and output data for relevant analyses (Xie et al. 2018). However, the shortcomings of this system include the following: (1) The geographic environment of the coal seam is not considered, and only the horizontal floor conditions are utilized. (2) The uncertainty in the interaction environment is less considered, and there is still a gap with real underground operational conditions. (3) There is insufficient real underground operational data to support the system design. Therefore, we develop a test method for a smart fully mechanized mining production system. The main framework of the proposed system is shown in Fig. 1.

Fig. 1
figure 1

Overall framework of the test and evaluation method

Full size image

The specific methods of the proposed schema are as follows. First, the full set of equipment and underground geological exploration data of a smart working face are integrated and fused into the FMUnitySim system to drive the virtual working face to run offline. Thus, the initial simulation data and virtual scene operation data are obtained. Second, the perception operation model is added to the virtual equipment, equipment, and geological exploration, while other means are input according to the parameters of the future smart development and operations to construct a virtual, AI, cooperative, operational simulation system. Finally, a comprehensive evaluation method that considers the cutting trajectory, straightness, workspace, and dynamic coal seam is developed. This could simulate the operations of fully mechanized mining equipment, determine the development trend, and test the operational performance of mining equipment.

Figure 2 shows the software design framework of the proposed system, where the Unity3D software is the core platform. The SQLServer stores the actual operational data of the integrated mining equipment, and Unity3D collects the virtual operational data. MATLAB then reads the actual operational and virtual simulation data from the SQLServer and Unity3D, respectively, and fuses them according to the error calibration parameters. The over-filtering and deep learning algorithms then return the processed data to Unity3D to update the virtual scene.

Fig. 2
figure 2

System software interactions

Full size image

3 Working face operational data acquisition and sensor parameter calibration

3.1 Smart mining and problem origin

We obtained some operational equipment and coal seam data in a working face from a coal mine located in China’s Shandong region. The smart level of this working face is high, and it adopts the technologies of shearer memory cutting, hydraulic support automatic following shearer moving, remote centralized control and manual intervention inspection, geological coal seam detection, etc. In 2020, the inertial navigation and longwall automation steering committee (LASC) was equipped. This allowed monitoring of the real-time operational status of equipment and coal seams in this working face (Fig. 3).

Fig. 3
figure 3

Images and demand analysis from the demonstration working face

Full size image

The director of the considered coal mine believes that the level of smart construction is not sophisticated enough, and there is a gap between this and existing mining robots. Thus, there is a strong need to further strengthen R&D. The proposed approach provides historic operational data for equipment, geological data, etc. The director has a clear future goal to evaluate the current operations, obtain the key limitation factors, upgrade the smart mining level to smart mining 3.0, and gradually approach smart mining 4.0.

3.2 Analysis and extraction of original equipment and coal seam geological data

The obtained data mainly include the shearer data, scraper conveyor data, hydraulic support data, and some coal seam information (Fig. 4). The volume is close to 150 MB in.csv format and contains cutting data for 100 cycles. After the analysis, the shearer data are complete with running position and altitude information. This provides accurate information in the 3D coal seam environment after algorithm processing. The positioning of the hydraulic support groups and scraper conveyor is based on the plane data as reflected by LASC. The exposure data of the two lanes of the working face are provided from a geological perspective, which includes the absolute geological elevation and discrete data from the five drilling points. However, information such as the roof graphic trend of the coal seam is lacking and can only be depicted indirectly by extrapolation through the overall data of the equipment operations and from fitting the complex surface with existing data to construct the roof and floor plates of the coal seams.

Fig. 4
figure 4

Relevant equipment and coal seam data for 100 cycles

Full size image

3.3 Future requirements of sensors and calibration of execution actions

The equipment performance and sensing control elements are calibrated, and the data in relevant papers with actual sensor configurations in the well are used to comprehensively determine the indexes based on the working principle of the sensors. For example, the positioning of the shearer is determined from three kinds of sensors: (1) The infrared sensor can only make rough judgments on which support the shearer runs to. The same value is given when the shearer both enters and leaves the same middle pan. Therefore, its accuracy is determined as a 0.8 support width (1.75 m), which is 1.4 m. (2) The encoder of the walking part has a higher positioning accuracy under ideal conditions of the working face. Under large fluctuations, the gap between the traveling wheel and pin row continues to accumulate, and the positioning accuracy is significantly reduced, which needs to be corrected. The average inclination of the working face under the mine is 4.5°, which is close to level. The accuracy is determined to be 0.5 m. (3) Inertial navigation is generally used as an altitude estimator to solve the three spatial angles of the shearer in real-time. Calculating the position and altitude of the shearer in real-time requires a comprehensive solution after a complete cycle rather than solving it in time, which has a delay. Thus, the mine configuration is an offline solution. Based on the existence of inertial navigation calibration and cumulative errors, its accuracy is determined to be within 200 mm. The shape accuracy of the scraper conveyor based on inertial navigation information and the relevant accuracy as inferred by the LASC system depend on the relevant performance of the inertial navigation. Therefore, this method has a lower accuracy than inertial navigation, which is calibrated to 300 mm. The accuracy of geological guarantee within 5 m in front of mining is approximately 1.5 m.

The mine currently still adopts the fourth-generation (4G) network. Considering all transmission links, the delay in acquiring sensor information is approximately 400 ms. The calibration information is confirmed to be effective based on the actual conditions. The director of this coal mine plans to gradually integrate a surface inspection robot (EXScan full mine-scanning system), drone scanning, and other new smart mining approaches. The accuracy of various equipment positions and altitude perception components could be improved, and video processing methods, such as camera imagery, can also be integrated. Further improvements are planned in all aspects of geological exploration means or equipment, sensing, and control elements. There are currently no accuracy indicators for visual monitoring and mine scanning, but ground-related indicators and the complexity of the mining working environment have expanded 1.5 times to 150 mm. If the 5G network is used, the delay time can be less than 200 ms. The data design of the virtual perception system is shown in Fig. 5.

Fig. 5
figure 5

Composition of the virtual perception system

Full size image

4 Offline scene construction-based running deduction

The virtual scene is constructed based on relevant running data. The data do not have direct 3D spatial coordinate information and cannot directly drive the operations of the virtual equipment. Thus, an initial high-precision scene must be established before the relevant derivation can be performed.

4.1 Offline operation and simulation system framework

As shown in Fig. 6, the architecture of the virtual offline operational system includes the data optimization-processing and virtual operation monitoring systems for mining equipment. The kinematic model between the equipment and coal seam allows the data optimization-processing system to utilize a series of algorithms and fuse the operational and virtual simulation data. The calibration parameters for the sensor errors provide the equipment operation data, which includes the shearer operation trajectory, cut-off trajectory, predicted cut-off trajectory, scraper conveyor altitude, and hydraulic support position. The virtual operation monitoring system establishes the initial virtual operation scene of the fully mechanized coal mining working face in Unity3D, which gradually improves the operation accuracy of the scene using the data provided by the data optimization-processing system. This improves the accuracy of the scene through continuous iterations and data optimization.

Fig. 6
figure 6

Offline operation simulation system

Full size image

4.2 Coordinate system and absolute altitude information of equipment

The absolute coordinate system of the virtual working face is built at the face-end hydraulic support of the transport lane and well-calibrated with geological information (Fig. 7). After establishing the absolute coordinate origin, the absolute geological information and equipment information could be further imported. This includes part of the drilling data and the exposed geological elevation of the two roadways. However, the geological information is significantly lacking.

Fig. 7
figure 7

Established coordinate system for the working face

Full size image

The kinematic models for the equipment and between the equipment and coal seam are established. This includes the coupling models of the shearer and scraper conveyor (Xie et al. 2017), scraper conveyor and hydraulic support (Li et al. 2021), cut-off trajectory and coal seam roof and floor plates, scraper conveyor and coal seam floor plate, and hydraulic support and coal seam, as well as the prediction model of the cut-off trajectory. The information on the equipment operations and coal seam gradually improved. As shown in Fig. 7, the operational data first drive the virtual shearer operations and provide the real-time 3D operations form of the shearer. The inversion of the scraper conveyor shape allows initially reconstructing the coal seam roof and floor. The height of the real-time up and down roller provides real-time corrections to the roof and floor plate with health status information. The current left and right cut-off tractions that represent the health state corresponding to each point in the bottom roll cut-off curve under the real-time cutting of the shearer are removed. If the data are within the normal range, those for the cut-off height are correct. Otherwise, the cut-off status is abnormal. Decreasing the roof plate height data and increasing the floor plate height data adjust the current to transform and correct the cut-off trajectory to the coal–rock partition interface.

The shape of the scraper conveyor as inverted from the shearer is fused with the plane shape information on the scraper conveyor in LASC. The rolling time-domain control method using the analysis and predictions is adopted to realize real-time sensing of the 3D form of the scraper conveyor based on the floor plate information as deduced from the previous step. Information on the hydraulic support is solved using information on the scraper conveyor through the floating connection mechanism approach. This is fused with the layout information on the support in LASC and the base inclination sensor of the base to solve the overall altitude of the hydraulic support groups in the absolute coordinate system.

4.3 Construction of coal seam simulation model in the working face

The coal seam trend of the fully mechanized coal mining working face is slow in the absence of geological formations. Thus, the equipment operation data between each cycle are similar to adjacent cycles to be mined in the future. Therefore, data related to real-time equipment operations and smart algorithms can be used to accurately model the limited transparent coal seam of the first few cycles. The geological model can be corrected to improve its accuracy. This provides operational and navigation information to the virtual equipment so that the accuracy of the coal seam is improved weekly, the equipment operation-working conditions are continuously transparent, and the trajectory of each cycle is fit to the roof and floor surface of the coal seam.

As the cut-off trajectories of the roof and floor are known, the three machine states and the position of the next cycle are predicted and judged based on the equipment positioning and setting method. We use the joint and cycle positioning and setting method of the three machines to extract the roof and floor plate curves as formed by multiple cycles of the cut-off from the front and rear drums of the shearer, the support state of each cycle of the hydraulic support, and the cycle layout state of the scraper conveyor. The fusion of the four datasets with two lines of the exposure data continuously updates the dynamic coal seam, describes its 3D state, and constructs the simulation model for the 3D working face. After determining the data, the coal seam point cloud is fit via parameterization and imported into Unity3D. The mesh of the coal seam floor, which is realized by connecting triangles, is divided and constructed. The mesh properties of the constructed mesh surface are added and rendered using the MeshFilter and MeshRender components in Unity3D to virtually reconstruct the coal seam based on the data (Fig. 8).

Fig. 8
figure 8

Construction method for the coal seam cutting floor based on a mesh grid

Full size image

4.4 Collaborative operational simulation method driven by offline operation data

The roof and floor surfaces of the coal seam as determined by the physical engine allow the equipment to run autonomously. The generated coal seam model reintegrates the equipment into the coal seam. The virtual operating system is driven by the physical engine and data to run collaboratively, which completes the construction of the initial scene. The hydraulic support group and scraper conveyor are connected through the floating connection mechanism and work together to move forward under the action of the coal seam floor and roof plates. The position and layout information of the support equipment on the coal seam floor provides for the virtual cooperative operations of the coal seam roof and support space of the hydraulic support group.

4.5 Workspace of equipment and dynamic coal seam analysis

The spatiotemporal kinematics for all equipment operations are studied through virtual operations to plan and analyze the relationships between the equipment cut-off, propulsion path, and coal seam as well as to optimize the path. Mending the key information points of the virtual equipment outputs the real-time coordinates in the extensible markup language (XML) format. The constructed spatiotemporal motion model can analyze the spatiotemporal kinematics of the equipment and coal seam. Considering an example of the supporting space of the hydraulic support, some key positioning points are arranged on the parts to obtain the entire state. These parts include the front and rear ends of the base, the four connecting rods, the front and rear ends of the roof beam, the guard plate, and the scraper conveyor near the coal wall side. The operational data are exported, compared, and analyzed with real data to determine the credibility of the virtual projections. The process includes predicting the cut-off trajectory, analyzing the straightness of the working face, reconstructing the working space, and real-time corrections of the coal seam to grasp all details.

5 Design and evaluation of virtual equipment cooperative operation simulation system

An accurate simulation model of coal seam operations is determined and a virtual equipment cooperative operation simulation system is established based on the virtual preview and data recording of the 100-cycle data in Sects. 3 and 4. The equipment could be set up with a degree of smart mining for accurate real-time analysis with changes in mining conditions.

5.1 Definition and division of smartness degree

Smart mining 1.0 is traditional manual mining, which has less sensing information. The decision-making and control depend on individuals, so these are not considered temporary. The definitions and characteristics of smart mining 2.0–4.0 are as follows.

  1. (1)

    The main feature of smart mining 2.0 is the “automatic control + remote intervention.” Based on the memory cutting of the shearer, automatic follow-up of the hydraulic support, and visual remote monitoring, the smart monitoring and centralized control of fully mechanized mining equipment are realized to ensure automated operations of coal cutting, pushing scraper conveyor, moving support, and transportation based on the program.

  2. (2)

    Smart mining 3.0 is characterized as automatic straightening. The imported LASC technology effectively controls the straightness of the working face. A geological model with errors that are within a certain range is constructed based on the transparent working face. The shearer relies on inertial navigation with a higher control accuracy and can realize normal operations of automatic production modes of the working face under complex conditions.

  3. (3)

    Smart mining 4.0 is the stage of fully smart and adaptive mining. The full smart mining strategy of “perception-analysis-decision-control” is developed through continuous deep learning based on the big data of smart mining under a transparent working face. Machine vision, multisource information fusion, and 3D physical simulations are used to analyze the collected data so the system can independently recognize and understand the state of the working face environment and equipment. Thus, the mining decision-making and execution control are performed by the smart decision-making control technologies, such as self-adaptive height adjustments to the shearer drum, straightness control, and up-and-down sliding control.

5.2 Operational framework of virtual mining equipment

The developed system realizes the above process in a VR environment, with the general framework shown in Fig. 9. Each piece of equipment should have the corresponding characteristics of perception–decision–operation capabilities at different stages. This is divided into the perception, decision-making, operation, and environment and posture change layers.

Fig. 9
figure 9

General framework for the AI operations of virtual mining equipment

Full size image

The perception layer is the interface between the equipment and the real world. It constantly reads the operational state and data to obtain information for decision-making. This behavior is realized primarily by the scripting interaction technology of the virtual simulation in Unity3D. The decision-making layer uses the results of the previous step to select the next appropriate behavior that matches the decision. This behavior searches the solution space and switches between multiple possibilities. Each virtual equipment should have a decision controller that selects relevant behaviors.

The decision results are used to issue commands, update the state, and perform virtual operations by the operation layer. The behaviors correspond to the virtual equipment simulation operations. However, attention is given to the consistency of the virtual and real behavior operations. The operational mechanism is analyzed by the environment and posture change layer based on the data obtained underground. The coal seam (roof and floor) and surrounding environment (gas and mine pressure) are also factors to manage the AI and evaluate the overall situation and posture of the cooperative operation system. The corresponding events occur based on the mechanism to make relevant evaluations of the overall equipment operations.

In general, the traditional agent relies primarily on a priori experience to model the environment. However, the complexity of the environment and tasks for the coal mining equipment and the uncertainty of the environment require relying on various means, such as sensors, for real-time dynamic modeling. A direct link between perception and action is established using sensing technology. Complex actions are then coordinated using the planning and decision-making of the sensors.

Analyzing the problems of perception, decision-making, and control in actual equipment operations indicates that the most typical problems are from the insufficient measurement accuracy of the sensors, delays in the network transmission system, error operations of the single programmed decision-making system, and lack of action for the control elements. Therefore, the issue is how VR can consider these factors for high-quality simulations. Relevant devices and buttons were added or disabled according to the user interface and different smart configurations. The basic framework is to utilize smart coal mining 2.0–4.0. The simulation begins after clicking OK; thus, the simulation process is visualized, and the running results are output. Finally, related decisions are made.

5.3 Virtual perception and information interactions

The information perception capabilities of AI equipment need to be designed based on the perception capabilities of real equipment operations. An AI may have multiple perceptrons, such as common sensor and environment (visual and auditory), which are used to detect dynamic equipment and environmental information. Perception cannot and does not need to be performed in each frame when the virtual AI is running, and the status of all other characters does not need to be queried. The perceptron system involves some complex computations, and the required information is obtained using two techniques: polling and events.

  1. (1)

    Polling obtains information by retrieving the surrounding environment and associated equipment, which is used in the existing centralized control system. This includes position and postural information about the shearer in real-time for comprehensive judgments on how to move the support group needs. A polling center is directly searched for variable interactions between objects to establish the virtual program.

  2. (2)

    Event-driven information is from sitting and waiting for messages. This can simulate the autonomous perception capability of equipment, including vision (image recognition), touch (hydraulic support and shearer collision beam), and hearing. The distributed control architecture adopts this approach. When the physical engine detects a collision, the trigger function is automatically called by adding a virtual perception component, and the central detection system (time manager) notifies all roles within a certain range.

Smart coal mining 2.0 is suitable for polling, smart coal mining 3.0 is close to an event-driven approach, and smart coal mining 4.0 is the combined driving mode of centralized and decentralized systems. While virtual sensors are noisy, the noise level is calibrated and needs to be reflected based on a certain rule. This is finally reflected in the perception accuracy. From smart coal mining 2.0–4.0, the amount of information required increases, which corresponds to how many variables are set in the virtual program. The reliability of information ranges from large to small errors, which corresponds to the relative accuracy of the perceptual variables in the virtual program. The reliability of the sensor networks influences the control decision-making process.

Different variables defined in the relevant scripts can be called directly in the virtual agent system, such as the position and altitude monitoring of the shearer, tilt monitoring of the hydraulic support, and displacement travel sensors. Considering an example from the data of the shearer inertial guidance system, the position of the shearer in the absolute coordinate system of the working face and the 3D altitude of the shearer are obtained as \((x_{zbc(i)}^{cg} ,y_{zbc(i)}^{cg} ,z_{zbc(i)}^{cg} ,\theta_{zbc(i)}^{cg} ,\gamma_{zbc(i)}^{cg} ,\phi_{zbc(i)}^{cg} )\). The real-time position and postural information of the shearer during actual operations can be obtained together with the relevant noise as calibrated by the sensors. In the future, improvements to the accuracy of inertial guidance can also be added using high precision and Gaussian distributions. In addition, the measurements can be entered automatically based on the actual positioning level of the current working face.

5.4 Virtual decision and control

The decision and control layers are designed after the perception layer, which completes the strategy for making relevant decisions and actions. The shearer, scraper conveyor, and hydraulic support are all integrated into a robot, and its agent model is established. The decision-making from smart coal mining 2.0–4.0 has experienced program control (direct data reading), semi-program control (reading and physical perception), and AI control (emotional integration). After the decision-making sends out relevant instructions, the actions of the control are paramount, which forms a closed-loop effect with the perception. After the scraper conveyor and hydraulic support are sensed, the ability to straighten them in place is influenced by many key factors.

The decision-making method for smart coal mining 2.0 is to control rules through the control program and make relevant decisions and actions. The shearer has a certain memory-cutting ability. The hydraulic support reads the relevant operational data of the shearer at any time and performs the actions of extending and retracting, moving the support, pushing, and sliding based on the set rules. These refer to a three-machine collaboration method (Xie et al. 2018). Coal mining 2.0 mainly performs relevant interactions by reading variables between various scripts. The perceived relevant information and calibration range allow adding a random quantity within the calibration error range to the variable, which indicates the perceived information is inaccurate. The time delay of the actual working face is expressed by obtaining the number of frames for the data delay. Control related to the transition is realized by considering the transition variable and adding a certain amount of noise.

The decision-making method for smart coal mining 3.0 has a certain degree of autonomy as well as comprehensively analyzing and determining the decision-making methods through a complex behavior tree. For example, the shearer can predict cutting. The relevant predictive cutting ability is embedded in the floor of the software. Compared with smart coal mining 2.0, the 3.0 not only has a rough map of the coal seam but can also identify coal and rock. Finally, it comprehensively decides the operational strategy of the shearer, and the control angle is more accurate, which refers to the cutting method. The straightness control of the working face includes the detection of the hydraulic support and scraper conveyor. The shape of the scraper conveyor is predicted based on the running track of the shearer, and the relevant straightening simulation is performed based on the LASC data.

Coal mining 4.0 integrates additional robots and smart elements, including for visual scanning and inspection. The inspection robot for visual-scanning data provides the posture of each hydraulic support through the relevant sensors between hydraulic supports. The collision and interference behaviors between the supports are then analyzed. The related models for reinforcement learning (RL) are also added to the decision-making and control methods. The distributed decision control method (Zhao et al. 2020) has no central node. An agent model is then established and upgraded to establish a deep RL agent model (Zaatari et al. 2020), which is given as:

 < A, S, R, P > 

Action space: A

State space: S

Reward: R: S × A × S → R.

Transition: P: S × A → S.

This is the classic quaternion in the RL; A represents all actions of the agent; S is the state of the world that the agent can perceive; R is a real value that represents the reward or punishment; and P is the world that the agent interacts with, also known as the model. The robots should decide the action based on the current environment and their state (Moss et al. 2021). These problems have the feature where smart mining needs to observe the environment and its state and then decide the action to achieve the desired goal (Peng et al. 2021).

5.5 Evaluation indicators build evaluation systems and methods to score

The AI equipment suggests that the described operation evaluation system for the fully mechanized coal mining working face is iteratively optimized and virtually previewed based on the results of the smart parameter analysis for the input equipment and geological exploration means. The equipment operational status and the influence factors for each parameter are evaluated based on the preview results. It is divided into major and minor factors, and a comprehensive evaluation strategy to operate the fully mechanized coal mining working face based on deep learning is established.

The goal of the three-machine collaboration is to continuously perform coal mining, transportation, and support operations both safely and efficiently. The effectiveness of the three-machine collaboration is reflected in the combination of benefits and the price to pay during the operational process. To control the three-machine cycle to complete tasks by comprehensively considering safety, productivity, and health, the comprehensive efficiency index J of the benefits and the price to pay are the optimization targets by adjusting the healthy state of the three machines at each stage.

The operational status assessments of the equipment include the shearer, predicted cut-off trajectory, scraper conveyor straightness, hydraulic support straightness, equipment-working space, and coal seam roof and floor plate. The operational status of the shearer is evaluated based on the parameters of the cutting section current and running speed of the shearer. The predicted cutting curve is evaluated based on the actual cutting trajectory. The straightness of the scraper conveyor and hydraulic support are evaluated based on the scraper conveyor and hydraulic bracket position parameters. The equipment workspace is evaluated by comparing the actual equipment workspace with the virtual equipment. The roof and floor of the constructed coal seam are evaluated based on the actual information about the roof and floor. The operating status of the subsequent fully mechanized coal mining equipment is adjusted based on the results of the equipment evaluations.

The constructed evaluation index model is shown in Fig. 10. The mining and transportation equipment is the cause of cooperative operations for a fully mechanized mining face, which determines the working face production efficiency and the cutting coal seam condition. The production efficiency includes the traction speed of the shearer and the load of the scraper conveyor. The cutting coal seam condition is evaluated from the coal retention, rock cutting, and health degree, which accounts for 30% of the total. The constructed virtual coal seam data are the benchmark of the coal seam. A comparison between the cutting roof and floor and the benchmark coal seam is used to judge the health conditions. The judgment is based on the coupling model of the shearer’s height adjustment operations and the coal and rock environment (Xie et al. 2018). The production efficiency and load of the scraper conveyor could be determined based on the coupling model of the shearer traction speed and the transportation volume of the scraper conveyor (Xie et al. 2018).

Fig. 10
figure 10

Evaluation index model for coal mining health and status

Full size image

The support and transportation equipment determine that the safe operation status of the working face accounts for a total of 30%. These are evaluated based on the safety efficiency (average following distance), maximum empty roof distance, straightness of the extracted relevant hydraulic support groups and scraper conveyor, and the working operational space of the equipment related to the health degree (mine pressure) of each support. These are determined from the coupling model of the follow-up control and shearer operation (Xie et al. 2018). The straightness of the scraper conveyor and hydraulic support are judged based on the method of extracting information from the chute or base of each support (Li et al. 2021). The health degree of support is determined from the coupling model of the follow-up control and the roof and floor (Xie et al. 2018). The overall operations account for 40%, which mainly includes the quality and effect of the coal production, shape of the formed dynamic coal seam, and the coal and rock cutting situation over the entire working face as solved in real-time with the working operation space of the equipment. These data can be directly read out from the virtual simulation software in 5.3 for relevant judgments. These factors are iterated and optimally solved to evaluate the overall operations.

6 Construction and testing of prototype system

6.1 Preview results of offline data

First, the virtual shearer is driven to run synchronously using the data, and the results for the roof and floor curves are plotted. The virtual scene is constructed by fusing the relevant data from the LASC scraper conveyor and hydraulic support group so it can run offline. If the test accuracy of the sensor element is ignored, the data for the run curve as derived from the virtual scene agree with the observed results. The accuracy of the posture and cut-off trajectory of the shearer operations is more than 90%, and the arrangement of the hydraulic support groups and scraper conveyor is more than 78%. The accuracy of the coal seam is fused with the data of the two lanes, which matches the geological report (Fig. 11).

Fig. 11
figure 11

Initial scenario construction and offline simulation results

Full size image

The cyclic data are predicted and compared with the data over 100 cycles based on the principle of similarity with satisfactory results. The coal seam corresponds to the cutting curve of the equipment. Real-time changes in the running state of the equipment and the shape of the coal seam are reproduced, and the prediction errors of the cut-off trajectory are less than 15% (Fig. 12).

Fig. 12
figure 12

Forecast coal seam conditions

Full size image

6.2 Analysis of offline data results and support workspace analysis

The accuracy of the inclination sensor and control element is ignored in the analyses. As shown in Fig. 13, in the VR simulation system (Fig. 13a), the working space of the shearer (Fig. 13b) is gradually constructed as the simulation system operates, i.e., the dynamic coal seam, working state trajectory of the scraper conveyor (Fig. 13c), and the support space of the hydraulic support (Fig. 13d). Their associated operating states and straightness are analyzed.

Fig. 13
figure 13

Evaluation method for the simulation workspace

Full size image

The roof and floor cutting curves, coal and rock cutting, and equipment operation space are derived and extracted at the 50th cycle via XML data. These are compared with offline data based on the shearer with an accuracy that meets the associated requirements. The real-time straightness information about the working face is derived, as shown in Fig. 14. The straightness information for the scraper conveyor and hydraulic support groups is extracted from the XML data as a 3D curve. The curve is projected in the work plane and compared with the data recorded by the LASC. We found that the two pieces of information correlate and meet the requirements for the straightness measurements of the working face. Therefore, the virtual working face constructed using this method could reproduce the relevant state of the equipment for offline operations.

Fig. 14
figure 14

Straightness information extraction of the scraper conveyor and hydraulic support groups

Full size image

6.3 Simulations of various conditions with different smart levels

The simulations are analyzed according to the smart coal mining 2.0, 3.0, and 4.0, and the design for the experimental scheme is shown in Table 1.

Table 1 Design of experimental scheme
Full size table

The simulation results for the three groups are shown in Fig. 14. To better observe the fully mechanized mining equipment workspace, all equipment is hidden after the workspace construction. The fully mechanized coal mining equipment workspace and the dynamic coal seam for the three cutting schemes are constructed. The black part represents the target cutting coal seam, and the pink represents the actual cutting coal seam. The red circles are the key information points of the shearer’s movement at every distance. As the analysis methods of the shearer’s cutting roof and floor are the same, the error analysis is performed using the roof as an example. The cut-off error of the shearer is expressed as the mean absolute error (MAE): \(\text{MAE} = \sum\limits_{i = 1}^{k} {\left| {Y_{i} - y_{i} } \right|}\), where \(Y_{i}\) represents the height of the key point of the cut-off path of the shearer and \(y_{i}\) indicates the height of the target trajectory. The equipment operation workspace and dynamic coal seam model are imported into the UG software for Boolean operations to determine the coal retention and rock-cutting quantity. The original 3D model is then hidden. In turn, the measurement volume is selected on the interface to measure the coal retention and rock-cutting volume. The coal retention and rock-cutting of the three schemes are shown in Fig. 15, and the obtained volumes are shown in Table 2. The other parameters are also derived.

Fig. 15
figure 15

3D schematic of the fully mechanized mining equipment workspace with three cutting schemes

Full size image
Table 2 Comparison of coal retention and rock-cutting volumes
Full size table

In the smart mining 2.0 cutting process, the shearer is manually operated to generate the demonstration cycle. The coal seam is mined completely based on the cutting height of the demonstration cycle. The high level of the shearer driver does not allow the formation of any rock-cutting volume. Therefore, manual cutting is conservative. There is no rock-cutting volume, and only coal retention volume is present (Fig. 15). In the smart mining 3.0 scheme, the coal retention is reduced, but the cut rock increases significantly. This indicates that geological detection methods and artificial smart mining algorithms should still be strengthened. The analysis shows that the autonomous cutting scheme of the shearer for smart mining 4.0 could cut more coal, produce a higher recovery efficiency, and improve production. This could cut the smallest amount of rock among the three schemes, which reduces the wear degree of the cutting teeth to a certain extent and increases the service life of the shearer. Thus, this is the optimal cutting scheme.

The scores are calculated based on the established evaluation system, as shown in Table 3. The accuracy of the cutting follows the order of smart mining 4.0, 3.0, and 2.0. The memory cutting in smart mining 2.0 has difficulty adapting to the complex and changing terrain. The partially smart cutting in smart mining 3.0 improves the accuracy to a certain extent, but there are large errors. The accuracy of cutting is significantly improved by autonomy in the equipment from smart mining 4.0. This schema provides the optimal cutting project.

Table 3 Experimental results
Full size table

6.4 Discussion

The proposed system opens a channel between actual mining data and virtual scene construction. The virtual operations of the fully mechanized coal mining working face are reproduced. The high accuracy, real-time position, and altitude relationship of the fully mechanized mining equipment in the coal seam environment are presented. The perceptual decision and control information in different smart stages are inputs for the simulations and evaluations, and the smart level of coal production is improved. The relevant use perspectives for coal production companies, coal machinery and equipment manufacturing companies, and scientific and technological management workers are given as follows:

  1. (1)

    For coal production companies, in the R&D process of fully mechanized smart mining, the overall improvement level is tested using various key technologies, such as perception, decision-making, and control. Relevant R&D plans should optimize the production capacity.

  2. (2)

    For coal machinery and equipment manufacturing companies, limitations in production equipment and the integration point between other equipment and geological environments are found, which promotes the R&D and design of smart equipment.

  3. (3)

    For scientific and technological management workers, the key technology and key points of coal mining equipment are found in the proposed system. The project will promote scientific and technological work in the coal industry and create breakthroughs in common key technologies.

7 Conclusions

The scene construction and offline simulations based on real data were realized in a virtual space. This reproduced the virtual operation-mining situations of a fully mechanized coal mining working face. Some predictive analyses of the future operating state were performed to realize the real-time accurate position–altitude relationship for equipment in the coal seam environment. Thus, the equipment operational information could be accurately determined to improve the mining automation level.

The entire life cycles of the fully mechanized coal mining equipment and smart mining technology are upgraded and transformed. From an overall perspective, the proposed system controls the design and operations of the working face as well as tests the relevance of each piece of equipment, sensing components, control components, and various key technologies. This provides analysis and evaluation for the current level of smart mining and the advancement of some local or small aspects of technological progress for the overall operations of the working face, identifies the key and necking points, and finds contradictions for the full attack and breakthrough. Therefore, coal production enterprises could establish a predictive smart fully mechanized coal mining concept and quickly promote smart construction.

The bridge between the performances of each piece of equipment, perception components, control components, and overall operations of the working face are established to test the equipment perception decisions. It is found that the operational capabilities could be improved. The surrounding environment is perceived by the virtual equipment, and a deep RL model is employed for the AI equipment analysis system to select the optimal strategy, take corresponding actions, and realize the cooperative operations of the fully mechanized coal mining equipment. Future work will realize collaborative linked optimization operations with online data and access to actual operational data for evaluations and analyses.