Abstract
This paper deals with the development of the content of Activity of Daily Life (ADL) based on VR (virtual reality) for cognitive function training for the elderly. The proposed ADL content has been developed focusing on a feedback technique based on performance analysis unlike in the existing VR-based cognitive training content wherein the customized training management based on performance evaluation was difficult. Experiments were conducted with 30 elderly people in the welfare center for 3 months. As a result of the effectiveness evaluation, the proposed ADL content showed higher results in all aspects of immersion, satisfaction, and performance. It is expected that user departure from existing ADL content will be prevented, and various artificial intelligence researches will be carried out to improve the technology accuracy to provide customized difficulty which is an important factor.
1. Introduction
As the average life span of people has been significantly prolonged, the incidence of geriatric diseases is rapidly increasing. In a keen competition environment, the reaction to excessive stress and anxiety felt by modern people stimulates and abuses the brain, causing a cognitive decline like the initial symptoms of dementia [1–4]. Modern medicine suggests coordinated motions based on exercise in a way that is effective for improving cognitive function which leads to the development of game content. Thus, it is necessarily important to develop game content which includes cooperative motions based on judgment, memory, reaction, and concentration which are representative cognitive functions related to dementia. Of the geriatric diseases associated with cognitive functions, degenerative brain diseases such as Alzheimer’s disease lead to the destruction of human nature unlike other physical diseases, and they are also considered as chronic diseases resulting in serious problems. Despite many studies, identifying the causes of such diseases can be difficult due to the complexity of the brain; hence, there are no clear treatments [5].
Thus, many patients and their families suffer severe pain after the onset of degenerative diseases, and the annual social and economic losses are enormous.
In Korea (according to National Health Insurance Corporation), an analysis of the expenditure on dementia patients showed that the number of patients with dementia in 2006 was 105,000 and increased to nearly triple in 2011 with 312,000. In other words, the number of patients with dementia increased by an annual average of 24.4% [6]. In 2025, the number of domestic dementia patients is expected to exceed 1 million [7]. In the past, the elderly with dementia were taken care of at home. However, due to social changes such as individualization and industrialization, the number of elderly people with dementia being taken care of at home is decreasing, and the number of elderly people requiring long-term care increases sharply as the elderly population continues to increase. That is, the number of elderly people with dementia entering long-term healthcare facilities will continue to increase. In addition, due to the nature of dementia, families have to suffer great pain as well [8, 9].
Recently, the cognitive and physical activities related to dementia are described in terms of Activity of Daily Life (ADL). As society ages more and more, the life of old age is changing into a trend on improving the quality of life. ADL is proposed as a concept of life activities, and it is necessary to develop ADL content that can enhance the quality of life. ADL content is mainly developed for the promotion of physical and cognitive functions and is being actively studied to maintain health in various life styles. In addition, ADL content improves the cognitive functions by raising judgment, memory, quickness, and concentration which are representative cognitive functions related to dementia [10, 11]. Functional dementia prevention games that allow elderly people to have fun and voluntary dementia-preventive training through the incorporation of fun elements of games are emerging as ADL content. However, because human cognitive, physical, and emotional characteristics do not develop independently, but they develop dependent on each other, the body’s motor skill and brain function should be improved altogether to help cognitive enhancement [12–15].
This paper provides a development of a program for improving cognitive functions and consisting of ADL content which is capable of increasing judgment, memory, and reaction for the elderly. The arm and finger movements were included to reflect as program input values. In addition, unlike the existing two-dimensional functional games, ADL content was developed under the VR (virtual reality) environment as a factor to enhance the user’s immersion feeling. The effectiveness of the ADL content was verified by continuous testing with 30 elderly persons in an elderly welfare center for 3 months. As a result, an improvement of the cognitive functions and satisfaction were derived and the validity of ADL content development was verified.
2. Related Works
2.1. Definition of Cognitive Function for ADL of the Elderly
The improvement of typical cognitive functions related to ADL for the elderly such as judgment, memory, reaction, and concentration is very helpful for the enhancement of cognition. The ADL content for the elderly must include elements that can improve such judgment, memory, reaction, and concentration. This ADL content evaluates the user’s brain functions and improves the cognitive functions through training and treatment as shown in Figure 1 [16–19].
The cognitive domain can be classified into 4 sections which includes memory, logical, attention, and reaction speed by mapping with the typical elements of the above-mentioned cognitive functions. All training content can be classified into subclasses based on the evaluation content. That is, attention is classified into “two-dimensional image rotation operation” and “attention and concentration,” while logical is classified into “logical solution” and “space-time location.” The reaction speed is classified into “reaction” and “boundary judgment,” while memory is classified into “impression memory” and “phase memory.” The cognitive, physical, and emotional characteristics of humans are not independent and they affect each other. In addition, using a game which merges the motions based on cognition and physical functions instead of mouse click or touch-based cognitive functions is more helpful in cognitive enhancement in order to prevent dementia more effectively. This paper deals with the development of ADL content using hand recognition in VR environment as an input value. The types of content are shown in Table 1 [20, 21].
2.2. Domestic and Overseas Trends of VR Content Development
There are various cases on the use of cognitive content both domestic and overseas. OMNI C&S, a smart healthcare-specialized company in Korea, introduced “OMNIFIT MindCare” and “OMNIFIT Ring” which are the first smart healthcare solutions that tend to the mental health in Korea. OMNIFIT MindCare provides a smart healthcare solution that can evaluate stress and brain mental conditions at a glance by measuring pulse and brain waves. In addition, A1 Future Technology plans to develop a “VR-based mild cognitive disorder awareness training system” that helps brain activation by using VR technology for medical use. The VR content enhances the training effectiveness by searching for things or ways in both indoor and outdoor environments. The game consists of shopping at the mart, finding items in the refrigerator, and cooking given disordered stages.
Oxford University researchers have used virtual reality (VR) to treat severe paranoia and made patients use the subway and elevator in a VR environment to demonstrate that VR is very effective in evaluating and treating mental health problems, showing applicability for major psychiatric problems. In addition, as shown in Figure 2, University College London (UCL) conducted an experiment to treat depression using VR in early 2016 by making the patients find out its causes and how to reduce depression symptoms such as self-accusation. Through this process, 9 out of 15 patients were able to treat depressive symptoms. In this regard, OMNI C&S has also provided content for promoting cognitive functions called OMNIFIT in Korea.
(a) OMNIFIT MindCare (OMNI C&S)
(b) VR content of the University College London
Although various types of cognitive training VR content have been researched and developed in Korea and overseas, they still lack factors such as minimum input and rules, continuous performance evaluation, and customized difficulty. In addition of course, most of the content may not be reliable because they lack clinical validation, which is an effectiveness evaluation. Motivation is the most essential element in healthcare. Neglecting the motivation element results in user departure and relevant content can become useless. Thus, a customized element is needed that becomes an important element that has been continuously studied in the field of cognitive training VR content. The proposed ADL content includes these elements and consists of appropriate cognitive training elements to enhance its effectiveness. In the next section, the step-by-step development of the content of the proposed ADL content will be discussed.
3. Development of VR-Based ADL Content to Improve the Cognitive Function of the Elderly
The purpose of this paper is to develop ADL content for the elderly and to improve the immersion of content usage and the effectiveness of cognitive learning in the VR-based environment. The whole content consists of cognitive games which can help improve cognitive functions such as reaction, judgment, and memory. The ADL content mainly consists of three mini games that can enhance the memory, reaction, and judgment. Each game is composed of the training elements of the cognitive enhancement content that suggest intelligence, learning, memory, thinking, and problem solving abilities for the cognitive characteristics of the elderly. In order to improve the cognitive function of the users, each game reflected these elements and made use of the content with hand gestures as input values to evaluate the cognition. Figure 3 shows the gameplay flowchart of the ADL content.
3.1. Development of User’s Gamification Model
The goals, rules, feedback system, and voluntary participation were emphasized in the configuration to personalize the ADL content and develop the customized game elements [22, 23] for the cognitive enhancement of the elderly. The initial ADL content was provided selectively at Levels 1, 2, and 3 (high, middle, and low), and Level 2 was set as default. Then, the degree of difficulty was adjusted through the analysis of the performance of the content, and the part of the game element was reflected on the content’s detailed composition. In the score evaluation, a differentiated relative scoring system was introduced to emphasize the elements such as competition elements and compensation. The user’s motivation was increased by assigning game points differently according to the perceived status or the degree of game performance. The overall content of this is shown in Figure 4, which shows that the overall ADL content is a customized guide-based service delivery.
3.2. Development Environment of Cognition Promotion ADL Content
3.2.1. Development of VR-Based ADL Content
The development of ADL content was carried out using Unity3D. Unity3D is a programming language that creates a virtual game environment by applying a physics engine. It models the game by linking the coordinate values of the user’s hand gestures with a 3D model and by implementing separate game logic and GUI. OpenNI Unity3D Wrapper was used as a plugin used in Unity3D. Therefore, the structure of the ADL content development system and Unity3D development structure can be identified by the content of each structure in Figure 5.
(a) Structure of OpenNI Unity3D Wrapper
(b) Structure of ADL content system
The OpenNI Unity3D Wrapper is a unity package file loaded into Unity3D for use by the program. OpenNI.Net.dll helps to use the OpenNI runtime library and supports its communication with the plugin. The OpenNI setting asset sets the runtime configuration, the player manager asset is used to detect or track the user, and the input controller asset reads the user’s hand gestures and transmits them to the player manager [24]. The overall system structure is largely divided into a user interaction, a game content, and user information layers. The game content layer can play the game based on the user input, perform the cognition and the body training, and give feedback on the content. It also checks if the motion recognition training has been executed and if the user can carry out the customized cognition training.
3.2.2. Development of Leap Motion-Based VR Interface
The interface module receives user input while using the ADL content and gives appropriate feedback. The VR interface applied to this ADL content is based on Leap Motion. Generally, elderly people who are subjected to ADL content are very unfamiliar with the input devices such as keyboards and mouse. NUI (Natural User Interface) is used as an input device for such low-skilled users, and the Leap Motion device is used as a supporting device for NUI. The examples and class structure of Leap Motion are depicted in Figure 6.
(a) Example of Leap Motion play
(b) Leap Motion Unity control class structure
The hand gestures such as pinch and grab in the interface are used as triggers, and the VR interface is designed with the Unity and Leap Motion interlocking structure. In the virtual environment, 3D objects related to the actions performed in the real world are arranged so that the user can perform familiar actions. Considering that most of the actual users are elderly, the interface supports user calibration function which treats the ADL content as completed even when the user did not complete the operation.
3.2.3. Function to Feedback the Degree of User’s Performance
The user feedback function analyzes the user’s ADL content performance and provides the appropriate level of difficulty. It divides the content into six levels of difficulty according to the score of the game. Difficulty adjustment enables the adjustment of cognitive elements provided in the ADL content. In a memory game, it becomes an increasing factor of information provided with time. In the reaction game, difficulty refers to the factor that reduces the speed and randomness of the flying object, while in the judgment game, it refers to a factor that lowers the difficulty of submitting a problem provided in the customized content of the user. The process of user performance feedback for providing such customized content is depicted in Figure 7. Generally, existing feedback researches have been using deep-learning-based methods [25–29]. In this paper, the method of setting the final difficulty level based on several conditions such as frequency has been applied.
It starts with the second stage, which is the level of difficulty for the initial user, and goes through the process of analyzing the performance. Body and cognitive status are measured based on performance analysis, and the level of detail is adjusted at Level 2 based on that information. If the performance is good, raise the level in the positive direction, and if the opposite, adjust the level of difficulty in the negative direction. The difficulty level adjustment is then adjusted based on feedback information one step at a time from Levels 1-1 to 3-2.
3.3. Development Result of Cognition Promotion ADL Content
The developed cognition promotion ADL content consists of 3 kinds of mini games that include the memory game, reaction game, and judgment game. First, the memory game enables the user to keep track of a given problem and provide the order of colors. Three color boxes will be shown for a certain period of time before it changes to question mark images. The user then puts the colors on the answer box, and the game checks if it is in correct order or not. The results of the final implemented memory game are shown in Figure 8.
Memory games are played in this process and are applied as a custom element with differential control of memory time. Since the initial play, the user’s performance has also been subject to feedback, in a way that increases or decreases memory time.
In addition, the reaction game enables the user to avoid flying bullets by jumping according to the timing, and the jump operation is processed by receiving the user input value through Leap Motion. Other user inputs include gripping the fist, enhancing the functions such as finger movement, and the quick action through the corresponding operation. Customized elements of the game start based on user performance analysis and are controlled by differences in the speed of incoming bullets. The results of the final implemented reaction game are shown in Figure 9.
Finally, the judgment game allows the user to pick up a color box that corresponds to the meaning of the letters written in the problem. The custom element of the game is the adjustment of problem difficulty, which is determined by the variety of color changes and the adjustment of time. Based on user performance analysis, the difficulty level is adjusted. The development and implementation results of the judgement game are shown in Figure 10.
4. Performance Evaluation
In order to evaluate the effectiveness of the proposed ADL content, a test population is organized and a subjective satisfaction survey on the population and an objective measure such as immersion and game performance were conducted. Satisfaction level was measured by using the recall scale after using the ADL content, while the level of EEG response was evaluated through an immersion test.
4.1. Experimental Environment
A pilot test with 30 elderly persons was conducted in a welfare center for 3 months. The performance evaluation of the ADL content consists of two methods, that is, by brain wave analysis and the evaluation of user performance on the content. In the EEG analysis (i.e., brain wave analysis), the level of concentration was analyzed through beta waves and the degree of involvement of the content was evaluated. The performance evaluation of content was compared with the results of ADL content gameplay, and the degree of improvement of cognitive ability was compared objectively. The summary of the experimental environment is shown in Table 2, while Figure 11 shows a concrete demonstration process.
The EEG device’s CPU was designed with an ARM M4 CPU. The hardware configuration is based on the ARM M4 core CPU, EEG sensor, RTC, LCD display, and BLE solution as shown in Figure 12. In the ARM M4 CPU block diagram, a 12-bit 200 kbps ADC capable of handling analog signals and 32 flexible GPIOs can be configured to use IOs such as various serial interfaces in a mapping scheme. Figure 12 shows the circuit design using the ARM M4 CPU block and ARM M4 core.
(a) ARM M4 CPU block diagram
(b) ARM M4 CPU schematic
The specifications of the EEG sensor device and actual device parts are shown in Figure 13.
(a) EEG device specification
(b) EEG device product
4.2. Effectiveness Evaluation
4.2.1. Immersion Evaluation of EEG-Based Program
In order to evaluate the proposed ADL content objectively, the content applied with the game model was compared with the content which was not applied with the game model through EEG analysis [30–32]. The most important element of the ADL content is the prevention of the program user’s departure. Thus, it is essential to provide a customized difficulty of content. As a result of this evaluation, the content which is applied with the game model and the content that was not applied with the game model were compared and analyzed through the EEG data. Figure 14 shows the subject status for EEG analysis and the sample values and concentration trends of the collected EEG results.
As a result, it was confirmed that the content which was applied with the game model showed a higher concentration by 17.7% on average in the middle and later parts of the gameplay. On the other hand, the content which was not applied with the game model showed a weaker concentration in the middle, and the results were gradually decreased to the latter half. This result can be verified sufficiently by looking at the graph of the EEG waveform shown in Figure 15. The EEG data set used in the experiment consisted of the waveforms of delta, theta, alpha, beta, and gamma, and each type of information was analyzed and evaluated. Some content could be less concentrated than the conventional content if inappropriate information will be suggested for the difficulty recommendation; however, it was verified that the ADL content reflecting the game model is highly usable.
(a) Collected EEG data sample
(b) Comparison of change in beta wave between content
4.2.2. Evaluation of Content Performance
For the performance evaluation of the content, the pre- and postperformance information of the proposed ADL content was analyzed [33]. The difference between the performance score on the first month and the score on the third month was used as a criterion to compare the problem solving ability of the content. This is a result of solving the problem that is configured for various cognitive function enhancements, and it is evaluated as evidence to indirectly demonstrate the improvement of cognitive functions. The difference in content solving ability between the first month (average 59.05 points) and the third month (average 75.35 points) can be clearly seen in Figure 16.
In order to analyze the satisfaction level of the content, a survey based on the Likert scale with 30 subjects with the same content with and without the game model was carried out. In addition, the user’s play time and score results were statistically analyzed, and the result of how the user truly performed the content was obtained. The satisfaction survey and performance results are shown in Figure 17.
As a result of the satisfaction experiment, the results of the proposed content were derived with many distributions of good and normal, and when the custom content was made with the general content without the custom element, it was not easy to adjust the appropriate difficulty level, resulting in many users leaving and unsatisfactory results. This allowed the effectiveness of the proposed content to be verified.
5. Conclusion
This paper deals with the development of ADL content for the cognitive function training of the elderly that increases their judgment, memory, and reaction. The movement of the arm (finger) was reflected on the program as input values, and the cognitive/motor component which is helpful for the enhancement of the cognitive function was applied. In addition, unlike the existing two-dimensional functional games, a user-customized gamification model and an appropriate user feedback function were added in the VR (virtual reality) environment to enhance the user’s immersion feeling. The validity of the ADL content was verified by continuous testing with 30 elderly persons in a welfare center for 3 months. Moreover, the validity to develop the ADL content was also verified by deriving the improvement of overall cognitive function and satisfaction. The most prominent feature of the proposed ADL content is the user feedback function, but it was found that the satisfaction level of the customized model becomes lower than that of the general content when an improper difficulty recommendation is made in the customized suggestion model. In order to improve the accuracy of the user-customized model in the future, it is essential to apply artificial intelligence (AI) technology in developing the recommendation process by reflecting the sensitivity factors for the difficulty control or learning the user performance data. In addition, we will consider developing cloud-edge computing technology based on IoT sensing by systemic expansion [34, 35].
Data Availability
The EEG signal data used to support the findings of this study are included within the article.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.
Acknowledgments
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. 2019R1G1A110034111).