Abstract
The turbidity interference caused by suspended particles in water seriously affects the accuracy of ultraviolet-visible spectroscopy in detecting water quality chemical oxygen demand. Based on this, the application of ultraviolet-visible spectroscopy to detect water quality chemical oxygen demand usually requires physical and mathematical methods to correct the spectral baseline interference caused by turbidity. Because of the slow response speed and unstable compensation effect of traditional correction methods, this paper proposes to use a compressed sensing algorithm to perform baseline correction and achieve good results. In the experiment, we selected formazin turbidity solution and sodium oxalate standard solution and carried out the research on the algorithm of turbidity correction for detecting chemical oxygen demand of water quality by ultraviolet-visible spectroscopy. The experiment obtains the absorption spectra of different concentrations of formazine turbidity solutions and the same concentration of sodium oxalate with different turbidity standard solutions at 210∼845 nm and analyzes the nonlinear effect of absorbance on turbidity. This article uses standard solution experiments to explore the compressed sensing theory for turbidity correction, and through the correction of the absorption spectrum of the actual water sample, it verifies the feasibility of the compression theory for turbidity correction. The method effectively corrects the baseline shift or drift of the water quality ultraviolet-visible absorption spectrum caused by suspended particles, while retaining the absorption characteristics of the ultraviolet spectrum, and it can effectively improve the accuracy and accuracy of the ultraviolet-visible spectroscopy water quality chemical oxygen demand detection.
1. Introduction
COD (chemical oxygen demand) is a parameter for calibrating organic pollutants in waterbodies, and it is an important index parameter for evaluating water quality [1]. Ultraviolet-visible spectroscopy is widely used in the field of water quality testing due to its fast, real-time, low maintenance cost, and no secondary pollution advantages [2]. The UV-visible spectroscopy method for detecting water quality COD has been studied by different teams. The team of Tianjin University Zhao [3] used the principle of ultraviolet spectroscopy to design and complete the online water quality COD detection instrument, which realized the real-time online detection of water quality COD. The team of Hou of Zhejiang University [4] used UV-Vis spectroscopy to integrate with other water quality detection methods to improve water quality detection performance. Ultraviolet-visible spectroscopy methods for detecting COD in water quality are gradually being updated, but in turbid waters, turbidity shifts the UV-Vis spectral curve of the water sample upward, and the shift is not proportional to the turbidity, which interferes with the direct measurement of COD using ultraviolet-visible spectroscopy [2]. Many scholars at home and abroad have carried out research on turbidity correction methods. Tang et al. [5] used a multivariate scattering correction method to effectively eliminate spectral shifts and shifts caused by turbidity scattering; Wu et al. [6] studied a turbidity interference compensation algorithm based on the Mie scattering theory to improve the soluble organic matter detection accuracy of the content. Chen et al. [7] proposed a method based on the first derivative to reduce the interference of turbidity and improve the accuracy of rapid detection of nitrate content by ultraviolet spectroscopy; Hu et al. [8] used normalization techniques to estimate the turbidity and simulate dynamically turbidity absorption spectrum, introducing attenuation coefficient for turbidity compensation. At present, the detection of water quality by ultraviolet-visible spectroscopy mostly uses multiple wavelength points or subbands for correction, which requires a large amount of experimental data support and cannot adapt to rapid changes in turbidity, and the compensation effect is unstable.
In recent years, compressed sensing theory [9, 10] has been widely used in the fields of signal processing, image processing, and data fusion. Compressed sensing is a new sampling theory. It uses random sampling to obtain discrete samples of the signal by developing the sparse characteristics of the signal under the condition of much smaller than the Nyquist sampling rate and then perfectly reconstructs the signal through a nonlinear reconstruction algorithm [11]. Compressed sensing usually includes three parts: sparse characteristics, measurement matrix, and recovery algorithm [12]. Based on the theory of compressed sensing, some scholars use compressed sensing for correction. Wang et al. [13] proposed a geometric correction method for compressed sensing to improve the correction quality of oscillation distortion in remote sensing images; Pan et al. [14] applied compressed sensing to improve the time resolution of the image; Duan et al. [15] used compressed sensing to correct distance migration to ensure the image resolution of compressed sensing imaging. Compressed sensing is numerically stable during the acquisition and reconstruction process. During the correction process, the factors that cause the offset change are certain, and it is very robust to the signal loss during the correction.
Based on this, this paper proposes the use of compressed sensing theory to correct the nonlinear rise of turbidity in UV-visible spectroscopy detection, select the standard solution of COD and turbidity, and analyze the influence factors of turbidity on COD in the whole UV-visible spectrum. The effect of turbidity on COD absorption spectrum is expressed sparsely, and the effect of turbidity on COD absorption spectrum is deducted from the original spectrum to realize turbidity correction.
1.1. Principle of the Experiment
1.1.1. Theory of Spectral Correction Based on Compressed Sensing
Turbidity affects the transmission of light through the sample, usually by slowly varying (red to blue) wavelength-dependent Mie scattering. Converting the transmission onto absorbance, this loss in transmission is manifest as a broad baseline, usually monotonically increasing from red to blue. Because of the influence of turbidity on the absorption spectrum, the measured spectral signal y is regarded as the COD absorbance spectrum x, and the turbidity absorption spectrum t consists of these two parts:
This paper proposes a method for turbidity correction of ultraviolet-visible absorption spectrum based on compressed sensing theory. In order to obtain the COD absorbance spectrum, a model is constructed as a formula:where is the observation matrix, which ensures that the turbidity influence matrix can be reconstructed effectively, and the observation matrix needs to satisfy the restricted isometry property (RIP) [16]:where , in which is an arbitrary K sparse signal, indicating that turbidity affects the absorption spectrum. If , the observation matrix is said to satisfy the K-order finite isometric property. For the sparse target in the transform domain, formula (3) can be expressed aswhere Ψ is the sparse base matrix, which contains the characteristics of turbidity on the absorption spectrum, and s is the sparse coefficient of the sparse transform domain, which is the turbidity influence factor.
After obtaining the turbidity influence factor, the turbidity absorption spectrum influence matrix is reconstructed through the L0 norm to solve the optimal solution:
To reconstruct the turbidity influence signal. Signal reconstruction algorithms have the following categories [17]: greedy iterative algorithm, convex optimization algorithm, reconstruction algorithm based on the Bayesian framework, and other algorithms. At present, the greedy iterative algorithm is widely used.
This article uses orthogonal matching pursuit (OMP) to achieve signal reconstruction. This method reconstructs the corrected signal very well and at the same time uses Gaussian denoising to process the noise of the signal and subtract the influence of turbidity from the original signal to obtain the corrected spectrum. The steps of the water quality COD turbidity correction algorithm based on compressed sensing are as follows: Step 1: obtain the turbidity influence matrix t from the original absorption spectrum and the absorption spectrum of the turbidity standard solution Step 2: calculate the observation matrix Φ and the sparse base matrix Ψ to obtain the turbidity influence factor s Step 3: reconstruct the influence matrix of turbidity absorption spectrum by solving the optimal solution of the L0 norm Step 4: the original spectrum deducts the influence of turbidity absorbance to obtain COD absorbance spectrum x
The turbidity influence matrix is determined by the ultraviolet-visible spectrum data of the formazin turbidity standard solution and sodium oxalate turbidity solution. For the full spectrum, the observation matrix and the sparse base matrix are obtained, and the turbidity influence data is sparsely represented, thereby determining the turbidity influencing factor under different turbidity factors. Under the turbidity influencing factor, the orthogonal matching pursuit algorithm is used to reconstruct the influence matrix of the turbidity influencing the absorption spectrum and obtain the turbidity influence in the actual measurement process. The influence of turbidity on the absorption spectrum makes different wavelength points increase irregularly with the concentration, finally deducting the influence of turbidity on the absorption spectrum from the actual measured full spectrum to obtain the COD absorption spectrum.
2. Methods
The principle diagram of the water quality detection system of ultraviolet-visible spectroscopy used in this experiment is shown in Figure 1.
The instrument used in the experiment has a DH-2000 deuterium tungsten halogen lamp light source, which can produce a stable continuous spectrum of 215–2000 nm; the optical path adopts ocean optics antiultraviolet fiber, which reduces the loss of ultraviolet light in the optical path and makes the ultraviolet spectral characteristics more stable. The fixed support for the water sample to be tested adopts the marine optical collimation support to ensure the stability of the light path. The photoelectric conversion module uses the Hamamatsu C10082CAH spectrometer, which integrates a polychromator with optical components, image sensors, and drive circuits. The typical spectral resolution is 1 nm. The spectrum collection and analysis are completed by the computer. This experiment was carried out under the same conditions. Considering the effective range of the ultraviolet-visible light source, the 210∼845 nm range was selected as the effective wavelength range, and a total of 1889 points were measured.
The experiment is divided into two parts: method research and method verification. This paper uses sodium oxalate standard solution to analyze the feasibility of compressed sensing in removing the interference of turbidity and uses modeling to analyze the spectral data of the actual water sample’s ultraviolet-visible spectrum, further verifying the feasibility and reliability of the method. The zero-turbidity deionized water used in the experiment was provided by the LD-UPW-VF comprehensive ultrapure water machine. According to “Water Quality-Determination of Turbidity” (GB13200-91), this reference suspension consists of a defined mixture of two chemicals, hydrazine sulphate and hexamethylenetetramine, to produce a “stock standard suspension” known as formazin. Prepare 400 NTU formazin turbidity standard solution, and dilute the 400 NTU formazin standard solution with deionized water to obtain 8 sets of different turbidity standards solution. Sodium oxalate and organic pollutants in water have the same reducibility, so sodium oxalate is selected as the simulated organic pollutant. Because this experimental study is aimed at natural waters with low organic content, a standard solution of 160 mg/L COD was prepared with sodium oxalate according to “Water Quality-Determination of Permanganate Index” (GB11892-89). The experiment selected the same amount of formazin and sodium oxalate solution mixed into 80 mg/L solution with different turbidity as the method research and demonstrated the feasibility of compressed sensing theory for turbidity correction and denoising. The accuracy of the method was verified by selecting a mixed water sample of a lake aqueous solution and different formazin solutions. All turbidity values in this experiment can be measured by HACH 2100Q after calibration. The COD value of lake water is 32 mg/L measured by Hach DRB200 and DR2800 instruments.
3. Results and Discussion
3.1. Standard Solution Experiment of Sodium Oxalate and Formazine
8 sets of turbidity standard solutions are measured with the turbidity of the 10, 20, 30, 39, 49, 61, 80, and 100 NTU, and the corresponding ultraviolet-visible absorption spectrum is obtained as shown in Figure 2. It can be seen from Figure 2 that, as the turbidity of the solution increases, there is an overall upward trend in the entire 210∼845 nm spectral region. Therefore, the higher the turbidity, the greater the impact on the absorption spectrum. For a stable turbidity solution, suspended particles in water scatter ultraviolet light, making the turbidity liquid have similar absorption characteristics, and as the absorption wavelength increases, the absorption characteristics gradually weaken and the absorbance decreases from the maximum value of 0.5∼4.1 to about 0.1. For the same wavelength, as the concentration of the standard turbidity solution increases, that is, the suspended particles in the water gradually increase, the more ultraviolet light is scattered, and the light absorption characteristics of the spectral curve gradually increase. The maximum absorbance value of the same turbidity near 210 nm increased from 0.5 to 4.1.
The above 8 sets of turbidity standard solutions are used and deionized water with a turbidity of 0 to mix the same amount of sodium oxalate solution with a concentration of 160 mg/L to obtain 9 sets of formazin and sodium oxalate with a COD concentration of 80 mg/L. For the mixed solution, the turbidity values of 9 groups of solutions measured by Hach HACN2100Q are 2, 7, 10, 15, 21, 26, 31, 42, and 51 NTU. The corresponding UV-visible absorption spectra are shown in Figure 3.
It can be seen from Figure 3 that the absorption peak of the sodium oxalate solution is near 214 nm, and the entire spectral region from 210 to 845 nm is affected by turbidity, which makes the entire spectrum appear nonlinear. Comparing the turbidity solution of the same concentration of sodium oxalate with the standard turbidity solution, the absorbance value of the sodium oxalate solution of 2∼51 NTU near 210 nm is in the range of 2.1∼4.1. The absorbance value of the turbidity standard solution of 10∼49 NTU is in the range of 0.5∼2.2. For the same concentration of sodium oxalate solution, as the absorption wavelength increases, the absorbance and turbidity standard solutions also show the same downward trend, and the absorbance value finally stabilizes at about 0.1. Therefore, it cannot be judged that suspended particles in water are not affected in certain bands of the full spectrum.
Therefore, in order to achieve accurate COD analysis, the effect of turbidity on the UV-visible absorption spectrum is effectively deducted. Using compressed sensing, the absorption spectrum of the mixed solution of formazine and sodium oxalate is corrected by algorithm, and the corrected ultraviolet-visible spectrum curve is shown in Figure 4. For the sodium oxalate solution of the same concentration, the turbidity influence factor is determined through the compressed sensing algorithm, and the influence of turbidity is deducted. Finally, the different turbidity solutions of 51 NUT∼7 NTU are corrected into an absorption spectrum curve.
Randomly select 3 of the above 8 groups of solutions as the test set, compare it with the low-turbidity spectrum curve with a turbidity of 2 NTU, and use the correlation coefficient R, root means square error (RMSE), and coefficient of determination R2 as evaluation standards to judge whether the compressed sensing correction model can be applied to the nonlinear correction of turbidity. The results are shown in Table 1.
The larger the correlation coefficient R after correction, the closer the correction spectrum and the low-turbidity spectrum line are, the better the correction effect. The RMSE represents the square root of the variance between the original spectrum and the corrected spectrum. The smaller the value, the higher the correction accuracy. The coefficient of determination R2 is used to measure the goodness of fit of the compressed sensing model. The larger the value, the higher the fitting accuracy. The evaluation standards are as follows:where xi is the corrected spectrum and yi is the reference spectrum, where and N is the length of the signal. It can be seen from Table 1 that the correlation coefficients of the three test sets are all above 0.98, and the root mean square error is less than 0.09. The compressed sensing model can be applied to turbidity correction.
Comparing the calibrated curve with the absorption spectrum curve of sodium oxalate solution without any turbidity measured at turbidity of 2 NTU, the correlation coefficient reached 0.9978, as shown in Figure 5. It is not difficult to see that after the turbidity of the ultraviolet-visible absorption spectrum of 51 NUT∼7 NTU sodium oxalate solution is corrected by the compressed sensing algorithm, the overall nonlinear rise of the spectrum caused by the turbidity is effectively corrected, and the characteristics in the ultraviolet absorption region are not weakened.
3.2. Actual Water Sample Verification Experiment
For an actual water sample of a lake, the COD value of 32 mg/L is measured by Hach DRB200 and DR2800 instruments. The instrument uses the permanganate method and is determined by chemical methods. The lake water and deionized water are used for comparative measurement, which meets the national standards. The turbidity measured by Hach HACN2100Q is 6 NTU, and the ultraviolet-visible absorption spectrum is obtained after turbidity correction, as shown in Figure 6. From the above experimental analysis, the overall effect of turbidity on the UV-Vis spectrum of water samples is a nonlinear rise, and it is difficult to use a single wavelength for correction. The compressed sensing algorithm is used to effectively realize the correction of the ultraviolet-visible absorption spectrum of the sodium oxalate solution with the same COD value and different turbidity solutions. It can be seen from Figure 6 that the actual low-turbidity water sample with a turbidity of 6 NTU is regarded as the reference group and compared with the corrected spectrum. The compressed sensing algorithm not only corrects the original ultraviolet-visible spectrum but also effectively guarantees the actual UV absorption characteristics of water samples. The correlation coefficient between the corrected spectrum and the actual reference spectrum reaches 0.9941, the root mean square error is 0.0098, and the model’s determination coefficient reaches 0.9447. The full spectrum can be calibrated well, which is helpful for the accurate calculation of water COD by UV-visible spectroscopy.
4. Conclusion
The sodium oxalate standard solution and the formazin turbidity standard solution were selected, and the turbidity correction method for the UV-visible absorption spectrometry detection of water quality COD was studied using the compressed sensing algorithm.
The results showed that turbidity shifts the curve of the UV-Vis spectrum of the water sample upward, and the amount of shift does not increase linearly. Therefore, turbidity affects the ability of UV-Vis spectroscopy to monitor water quality COD. The compressed sensing algorithm proposed in this paper can effectively realize the turbidity correction of the UV-visible absorption spectrum of the actual water sample without affecting the UV-visible absorption characteristics of the UV-visible spectrum. The correlation coefficient of the actual water sample reaches 0.9941, and the root means square error is 0.0098.
The water quality UV-visible spectrum data correction method based on compressed sensing provides a new method for accurately and effectively realizing the accurate calculation of water quality COD and provides a theoretical basis for further realizing the rapid determination of water body parameters.
Data Availability
This article contains floating-point data used to support the results of this research. The floating-point data used to support the results of this study can be obtained free of charge from https://drive.google.com/file/d/1GLz070TpOgkF7-2_-bdXA2z8_F8MH3Il/view.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This research was supported by the National Natural Science Foundation of China Youth Fund Project (Project no. 61805029) and General Project of Chongqing Natural Science Foundation (Project no. cstc2020jcyj-msxmX0879).