Abstract
For mobile robots and location-based services, precise and real-time positioning is one of the most basic capability, and low-cost positioning solutions are increasingly in demand and have broad market potential. In this paper, we innovatively design a high-accuracy and real-time indoor localization system based on visible light positioning (VLP) and mobile robot. First of all, we design smart LED lamps with VLC and Bluetooth control functions for positioning. The design of LED lamps includes hardware design and Bluetooth control. Furthermore, founded on the loose coupling characteristics of ROS (Robot Operator System), we design a VLP-based robot system with VLP information transmitted by designed LED, dynamic tracking algorithm of high robustness, LED-ID recognition algorithm, and triple-light positioning algorithm. We implemented the VLP-based robot positioning system on ROS in an office equipped with the designed LED lamps, which can realize cm-level positioning accuracy of 3.231 cm and support the moving speed up to 20 km/h approximately. This paper pushes forward the development of VLP application in indoor robots, showing the great potential of VLP for indoor robot positioning.
1. Introduction
With the development of large-scale facilities such as underground parking lots and shopping malls, robots are required to do more difficult and intelligent work, and the application scenarios of robots become more complex and diversified [1]. In order to improve the performance of the robot in the working process, precise and synchronous localization system for indoor settings is absolutely essential at present. Obviously, one of the key problems of mobile robots is how to enable them to learn the ability of autonomous navigation, which requires a more accurate and simultaneous positioning system. Previous research on robot positioning has mainly focused on high-cost external sensors such as cameras, depth-sensing cameras, laser rangefinder, and complex algorithms [2–5], which require more computation to achieve higher accuracy. WiFi-based positioning is also a common solution for indoor positioning, but even if the data of multiple access points are fused with particle filter, the positioning error is still more than 1 m [6].
As a wireless transmission technology, visible light communication (VLC) uses electrical signals to control the high-speed flashing LEDs to transmit information since the photosensitive device can detect the high-frequency flicker and restore it to the information to be transmitted. VLC technology not only has rich spectrum resources but also suffers little external interference, which can expand the spectrum of the next-generation broadband communication technology. Based on indoor VLC technology, VLP (visible light positioning) can realize indoor positioning by sending position information of LEDs to the positioning terminal. VLP can provide convenient data services for indoor users anytime and anywhere since visible light is closely related to people’s daily life and various places contain visible light source, for which VLP is superior to other positioning technologies in terms of hardware cost investment and portability. The positioning system by radio has limitations on use due to the generation of acoustic interference to indoor electronic equipment [7], while VLP has high real-time, free of electromagnetic radiation, and can be applied to many scenes, such as hospital and nuclear plant. For example, nuclear power plants require robots to work in high radiation, high temperature, and high pressure environment, for which VLP is well suited because of its high transmission rate and anti-interference capability. Furthermore, it has better antimultipath ability in the indoor environment, which makes it possible for the visible light location to provide higher accuracy [8].
In general, VLP broadly falls into two categories by receiver type, namely, the photodiode-based and the camera-based. Because of its high sensitivity to light, photodiode (PD) can separate the light source in space, and the positioning accuracy is not affected by the interference of surrounding light [9]. However, the positioning method based on PD will cause large errors due to angle measurement, received signal strength measurement, light intensity change, and other reasons, resulting in poor positioning effect [10]. Therefore, some work has been done to overcome this shortcoming. In [11], visual information captured by the camera to estimate the incidence angles of visible lights first is utilized, and the visual and strength information of visible light signals are combined to improve the effect of location. Reference [12] uses multiple PDs and machine learning, which enable the system to localize accurately with two visual luminaires. Although these efforts optimize the PD-based VLP system to some extent, they require the addition of additional sensors. By contrast, camera-based has higher stability and anti-interference ability and more commercial potential as high-demand image sensors for commercial terminals. Several prototypes of smartphone-based VLP systems have been developed [13–16], but these systems yield relatively modest performance, usually not enough for practice. The proposed VLP system with commercial smartphones in work [17] can support moving speed up to 18 km/h. The work in [18] used at least three LEDs to transmit their three-dimensional coordinate information, which was received and demodulated by two image sensors near the unknown position and the unknown location is then calculated based on the geometry of the LED image created on the image sensor. An angle of arrival localization algorithm based on three or more LEDs was proposed in work [15], where a camera is regarded as an angle-of-arrival sensor and its average error is >10 cm. The work in [19, 20] uses EKF-based fusion method to realize robust visible light positioning system with an IMU and a rolling-shutter camera, which can reach centimeter-level accuracy. The work in [10, 19, 21] first proposed VLP-based system on robots; however, they all implemented the system at the experimental level only.
Therefore, in this paper, we set up a robot localization system in the office with high-accuracy and real-time positioning, that is, an indoor robot positioning system based on the robot operating system (ROS) and VLP, which realizes the accuracy of 2.14 cm and support the moving speed up to 20 km/h. It combines ROS and VLP and puts into practice. In addition, we design the smart LED lamps with VLC function for positioning that can be controlled through Bluetooth. The rest of the paper is organized as follows. Section 2 introduces the design of VLP-based robot system. Implementation and analysis of the VLP-based robot system is discussed in Section 3, and Section 4 is the conclusion.
2. The Design of the VLP-Based Robot System
The architecture of the VLP-based robot system is as shown in Figure 1, and the smart LED lamps with VLC and Bluetooth control functions are used as transmitter for positioning information and illumination. The camera installed vertically on the robot captures image sequence, and then through the VLC information obtained by dynamic LED-ROI tracking algorithm and LED-ID recognition algorithm, the robot uses the location algorithm to locate its own position.
2.1. The Design of Smart LED Lamp
The VLP takes the LED lamps as the signal hotspot to send the location information. LED lamps are driven by VLP modulators that simultaneously illuminate and broadcast their location signals. When the LED lights of the VLP modulator are installed, each lamp is assigned a unique identifier and the installed LED lights are associated with the corresponding coordinates in the positioning system database. As the source of location information for the positioning system, a LED lamp with visible light communication function that can be adjusted through Bluetooth is described in this section.
2.1.1. Hardware Design of Smart LED Lamp
The self-design smart LED lamp is provided with VLC function with the Bluetooth cloud control model, which makes it easy to change ID code and location-based monitor. By using visible light as the signal transmission carrier, visible light communication equipment can be used as both illumination device and signal source, greatly reducing the cost of equipment. The hardware design of smart LED lamp is shown in Figure 2, and VLC controller is added between the power supply (LED driver) and the LED lamp. Using the BLE SoC as the microprogram controller of the controller, VLC data/frequency can be configured directly through the BLE wireless channel. VLC control signal can be generated through one of the I/O pins of BLE SoC.
2.1.2. Generation of Modulation Signals Based on Bluetooth Control
As shown in Figure 3, the hardware architecture of a common BLE SOC consists of the following components: many SOCshave DMA controller especially, which can transfer data from peripheral devices such as ADC to memory with minimal CPU intervention, thus achieving high overall performance with high power efficiency.
The universal serial peripheral interface (SPI) with DMA function is used in the proposed system instead as shown in Figure 4. DMA controller is set to repeat mode, and SPI frequency is adjusted according to VLC bit duration. After being powered on, all stored VLC data are loaded from flash memory to the dedicated RAM area and its starting address is configured as the source address of DMA controller. After triggering, the SPI module will start the continuous VLC transmission without CPU intervention [22].
The basic architecture of smart LED lamps with Bluetooth control is shown in Figure 5. Information about each lamp’s VLC identifier (ID) and its Bluetooth address, as well as its physical location, is stored in the cloud [23]. The position data is uploaded by Bluetooth and stored in the on-board data memory in the VLC controller. Then, the memory generates a modulation pattern to modulate the LED lamps and broadcasts a unique position identifier provided by each lamp, breaking through the key points of indoor localization. Users can shoot the LED light through the camera on a smartphone to obtain the unique VLC ID corresponding to the LED and then convert the received VLC ID into the corresponding Bluetooth MAC address stored on the remote server or cloud and realize the further control of LED lamps. It also can get the actual location of the LED, which can be used as data for subsequent visible light indoor positioning in the proposed system.
2.2. Rolling Shutter Mechanism of the CMOS Image Sensor
Based on the modulated LED lamps, the recognition of LED-ID is realized by using the image sensor-based VLP, which utilizes the rolling shutter mechanism of the CMOS image sensor. All pixels on the sensor are exposed at the same time as the CCD sensor, so at the end of each exposure, the data for all pixels is read out simultaneously. This mechanism is often referred to as the global shutter for the CCD sensor. However, for a CMOS sensor, when exposure of one row is completed, the data of this row will be read out immediately, which means that the exposure and data reading are performed row by row. This working mechanism is called the rolling shutter mechanism of the CMOS sensor. The LED image captured by the CMOS sensor would produce bright and dark stripes while turning the LED on and off during a period of exposure due to the rolling shutter mechanism of CMOS sensor. As shown in Figure 6, the LED image acquisition using CMOS sensor is illustrated.
2.3. The Region of Interest (LED-ROI) Area Tracking Based on Improved CAMshift-Kalman Algorithm
As far as we know, real-time positioning of mobile robot requires real-time shooting and processing of each image to obtain the region of interest (ROI) area of the LED luminaire in the image. The success of the LED-ID detection and recognition method is inseparable from the accurate detection of LED-ROI, which determines the real-time performance and robustness of the system. We use an improved CAMshift-Kalman algorithm to improve the accuracy and robustness of the VLP-based system. In order to obtain better tracking performance, the Bhattacharyya coefficient was used to update the observation noise matrix of Kalman filter in real time. The CAMshift algorithm is used to track the output position of the target as the measurement signal, and Kalman filter algorithm is used to correct the target position. The algorithm not only combines the CAMshift algorithm with the Kalman filter but also introduces the Bhattacharyya coefficient. For more details, one could refer to our previous work [24]. We tested the effect of the algorithm in the dynamic case with modulator tubes as background interference. Under the interference of LED tube, this algorithm can still ensure the accurate detection of LED-ROI, which reflects robustness and good real-time performance of the algorithm. The dynamic tracking performance of the improved CAMshift-Kalman algorithm is shown in Figure 7.
(a)
(b)
(c)
(d)
2.4. The LED-ID Recognition
The LED-ID recognition in this paper is to give certain characteristics to the light and dark stripes captured in the image. By introducing four characteristic variables, frequency, duty cycle, distance, and phase difference coefficient, the characteristics of LED-ID optical strip code captured by CMOS image sensor are given. The characteristic can be the number of light stripes of the stripe code in the LED pixel area, the area of the LED pixel, the width of the bright stripe is greater than the width of the bright stripe and the width of the dark stripe, and the phase difference coefficient between the stripes. Then, after the LED-ROI was obtained by the aforementioned algorithm, these features are extracted through simple image processing technology, and the location information of LED-ID is recognized through the preestablished database. The details could be found in [25].
2.5. Triple-Light Positioning Algorithm
As shown in Figure 8, when three or more LEDs are detected in the vision of the image sensor, the robot selects three LEDs for positioning after information extraction through the algorithm. The coordinates of the LEDs are (x1, y1, z1), (x2, y2, z2), and (x3, y3, z3). Generally speaking, the height of the ceiling is the same in one place, so z1 = z2 = z3. The position of the center point O of the lens can be calculated by the similar triangle relationship.
Therefore, according to lens focal length and the coordinates on the image plane (, ) , the distance between each LED image center and lens center can be calculated:And according to the similar triangle, the distance from the center of the lens to the LED anchor can be obtained:where H is the vertical distance between the LED and the lens plane.
Then, the coordinates of the lens center point (x, y, z) can be calculated by the following formula:
The aforementioned formula can be converted toAnd the z coordinate is
According to equations (5) and (6), we can calculate the position of the camera lens center . Meanwhile, the coordinates of the robot can be calculated through the static conversion relationship between the camera and the robot base. The analysis of the algorithm could be found in our previous work [26].
3. Implementation of the VLP-Based Robot System
3.1. System Setup and Implementation
The system setup of the VLP-based robot system is shown in Table 1. Mobile robot (Turtlebot3) was used to build the VLP-based system platform. The images of LEDs were shot by MindVision UB-300 industrial camera which is fixed vertically on the robot by prior extrinsic calibration and transmitted by Raspberry Pi 3 Model B. As shown in Figure 9, we implemented the system in the office with three smart LED lamps installed on the ceiling for positioning. The related information of LED lamps is shown in Figure 9. In positioning process, the image sensor can detect at least three LEDs to ensure the realization of the triple-light positioning Algorithm. And for the weak processor performance of Raspberry Pi 3 Model B, the program of image processing and location calculation is run on a remote controller. The operating system of the Turtlebot3 robot is Ubuntu 16.04 MATE which corresponds to the Kinetic version of ROS, and the system of the remote controller is Ubuntu 16.04 desktop. We demonstrate the performance of the VLP robot system on ROS. The camera installed vertically on the robot captures the image and then obtains the location information stored by the LED immediately through the LED-ROI dynamic tracking algorithm and LED-ID recognition and finally realizes the centimeter-level location through the triple-light positioning algorithm. Purple dots are used to represent the positioning result of the geometric center of the camera obtained by VLP.
3.2. Positioning Accuracy of the VLP-Based Robot System
In order to evaluate the positioning accuracy of the system, two series of experiments were carried out. The first series was used to test the performance of the stationary robot. As shown in Figure 10, uniformly distributed points in the experimental area were randomly selected to calculate the standard deviation between the measured position and the actual position. For the positioning accuracy shown in Figure 10, 90% positioning errors are less than 3.231 cm, the maximum positioning error is no more than 5 cm, and the average positioning error is 2.14 cm.
(a)
(b)
(c)
(d)
We test the performance for moving mobile robot in the second series of experiments, and we control the robot to go straight and travel a distance after turning to demonstrate the real positioning effect of the localization system with different speed. As shown in Figure 11, the measured positions of the robot are plotted as purple dots and the robot industrial camera are highly coincident with the positioning results in different motion states in the refined section of the robot, which reflects the good positioning effect of the VLC positioning system on ROS mobile robot. Due to the restrictions of ground field, we let the robot travel at the speed of 2–4 cm/s. However, the moving speed of the robot has no effect on the positioning accuracy but depends on the real-time positioning which is discussed in Section 3.3. It is worth mentioning that the position results in the coordinates of the center of the camera fixed vertically on the robot, which can be transformed to the center of the robot by coordinate transformation in the VLP robot system.
3.3. Real-Time Positioning Speed of the VLP-Based Robot System
Location speed is another key factor in the VLP-based robot system, which represents the mobile robot’s moving speed when receiving VLC information and calculating the current position in time. Therefore, for the IS-based VLP system, before the mobile robot passes through the VLP information transmitter, it needs to capture and extract the VLP information, that is, before the VLC light leaves the field of view (FOV) of the CMOS image sensor. Therefore, the maximum supported moving speed is the speed at which the mobile robot extracts the VLP information over a period of time from the edge of the leftmost to the right. We assume that the target LED is tangent to the left edge of the image in frame N and to the right edge of the image in frame N + 1, as shown in Figure 12, and the maximum velocity of the mobile robot is defined as , where is the distance the terminal device moves between two frames, and is the time from the beginning of the image capture to the completion of the positioning calculation. Based on the proportional relationship between image coordinates and world coordinates, the relation is expressed as , where r is the pixel length of the image, is the actual diameter of the LED, and D is the diameter of the LED in the image.
(a)
(b)
As mentioned in Section 3.1, due to the weak processing capacity of Raspberry Pi 3 in Turtlebot3, most of the image process cannot be directly performed on Raspberry Pi. Therefore, the images captured by the CMOS camera in the robot are transmitted to a remote laptop for processing, leading to positioning delays. We set the timer and counter in the program, and the average time t is 0.3893 s. As can be seen from Table 1, the actual diameter of LED is 150 mm. The pixel length of the image is 800 (pixels), and the LED diameter in the image is 55.54 (pixels). According to the definition of speed, the maximum movement speed of the positioning terminal is 9.497 m/s; ≈ 20 km/h. If the robot is equipped with a small computer with better computing power, it can save the time it takes to transmit data and support higher moving speeds.
4. Conclusion
In this paper, we design a high-accuracy and real-time VLP-based robot system on ROS and implement the indoor VLP system based on TurtleBot3 robot platform in the office as a practical application scenario, which is equipped with designed LED lamps having VLC and Bluetooth control function. The implementation of the system shows that the positioning accuracy of the VLP-based system can reach cm-level position accuracy of 2.14 cm and have a good real-time performance that can support the moving speed up to 20 km/h, which has a great research prospect in the autonomous positioning and navigation of indoor robots. In the future, we will apply the VLP-based system to the fusion localization or robot navigation of ROS, making full use of the advantages of VLP while making up for the defects of visible light positioning.
Data Availability
The source code of our finished article is not open to the outside world, and the experimental data have been reflected in the Experimental Analysis section.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Authors’ Contributions
Xianmin Li contributed to the conception of the study, performed the data analysis, and helped perform the analysis with constructive discussions. Zihong Yan contributed significantly to the analysis and manuscript preparation and wrote the manuscript. Linyi Huang performed the experiment and analysis with constructive discussions and wrote the manuscript. Shihuan Chen helped revise the manuscript. Manxi Liu helped perform the analysis with constructive discussions. Xianmin Li, Zihong Yan, and Linyi Huang contributed equally to this work.