1.

Introduction

Hyperspectral imaging (HSI) has been used in many fields of biomedical research and clinical applications in recent years.¹ Promising results were reported for the estimation of organ perfusion,^2,3 differentiation of anatomical structures,^4,5 wound assessment,^6,7 and the detection of cancerous tissue^8–11 in animal experiments and clinical studies.¹² However, the majority of the described systems do not apply to current minimally invasive surgery. Supporting the estimation of tissue perfusion or detection of risk structures with spectral imaging would be particularly important in laparoscopic procedures where surgeons are limited to their visual impression.

Reported multi- and hyperspectral endoscopic systems since 2013 are listed in Table 1. Two systems are capable of investigating both the visible and near-infrared range, but are limited by long acquisition times and without color video during spectral imaging or low spectral resolution.^13,14 Simultaneous hyperspectral data and color video acquisition in the visible range were achieved with pushbroom and manual line-scanning flexible endoscopes.^15,16 Most systems are using spectral scanning methods like acousto-optical tunable filter (AOTF), liquid crystal tunable filter (LCTF), filter wheels, or light sources with monochromators. Spectral scanning typically results in high spatial resolution but longer acquisition times per spectral channel and misalignments due to motion. Spectrally resolving detector arrays (SRDA) or snap-shot imagers are nonscanning systems and are, therefore, not limited by motion artifacts.¹³ However, spatial and spectral resolutions are reduced. Spatial or pushbroom scanning allows high spectral and spatial resolution at the same time. Previously described endoscopic pushbroom systems have the disadvantage of having to move the sample or the endoscope itself.^14,16

Table 1

Specifications of reported multi- and hyperspectral laparoscopic (HSI/MSI) systems since 2013.

Here, the presented laparoscopic system provides a high-resolution color video and simultaneous HSI with high spatial and spectral resolution in the visible and near-infrared range. To the best of our knowledge, it is the first system with these capabilities. Furthermore, the typical limitation of pushbroom scanning is overcome for the first time in a visible and near-infrared laparoscope. Moving the spectrograph with a miniaturized stepper motor inside the housing of the laparoscopic camera allows HSI without moving the sample or the laparoscope. This paper aims to describe an HSI laparoscope with high practicability for clinical use and to evaluate it with different light sources. Additionally, HSI was performed on a human tissue resectate with the presented system and a previously described hyperspectral camera for open surgery for comparison.¹⁷

2.

Methods

The HSI laparoscope was evaluated in a laboratory set-up with different light sources and variable working distances to the measured reference objects. Furthermore, data from human tissue resectates were acquired with the proposed system and a commercially available HSI system for comparison.

2.1.

Description of the HSI Laparoscope

The overall system is composed of an HSI and color video camera system combined in a three-dimensional (3-D) printed housing with dimensions $8\times 6\times 6{\mathrm{cm}}^3$ and a rigid laparoscope (HOPKINS^® 8711 AGA, KARL STORZ SE & Co. KG, Tuttlingen, Germany) with a 10-mm diameter suitable for white light, near-infrared fluorescence, and autofluorescence imaging. The laparoscope is attached to an objective lens (OL) adapter with a focal length of 14 mm and a C-mount connection on the camera side. The weights of the camera and the used laparoscope (with OL adapter) are 259 and 254 g, respectively. For illustration of the miniaturized camera system components, see Fig. 1. A beam splitter (BS) is partially reflecting the incident visible light only, while the remaining light is transmitted. The reflected light is captured by a color image sensor (IMX290LQR-C, Sony, Tokyo, Japan) with $1920\times 1080$ maximum number of effective pixels at 120 frames per second (fps). After the transmitted light went trough a high-pass filter glass (FG) (GG 495, Schott AG, Mainz, Germany), only light above 500 nm is passing the entrance slit of the spectrograph. Two achromatic lenses are used for focusing and a transmission grating splits up the light that is captured by a monochrome camera sensor (IMX290LLR-C, Sony, Tokyo, Japan). This sensor has a relatively high quantum efficiency in the near-infrared range and can capture 120 fps with full high-definition (HD) resolution. The electronic components of the camera are controlled by a system on module (i.MX8M Quad 1.5 GHz ARM Cortex-A53, NXP Semiconductors, Eindhoven, Netherlands) and the acquired raw images are transferred via gigabit ethernet to a Windows machine (2.11 GHz Intel Core i7) for processing. During scanning, $960\times 780\text{pixels}$ ($\text{spatial}\times \text{spectral}$) of the maximum $2048\times 1088\text{pixels}$ of the image sensor are read out at 140 fps. After spectral calibration of the spectrograph with a krypton lamp once, each of the 780 pixels in the spectral dimension can be assigned to one of 500 channels between 500 and 1000 nm. This $960\times 500$ raw image is binned to $480\times 100$ ($\text{spatial}\times \text{spectral}$) pixels for noise reduction by averaging two pixels in the spatial dimension and five pixels in the spectral dimension. A detailed description of the used calibration and balancing process can be found by Holmer et al.¹⁸ The spectrograph is guided by a linear rail to ensure the axis alignment while being moved by an attached stepper motor with a diameter of 15 mm. For the acquisition of a hyperspectral data cube, the spectrograph is moved over the image plane by the stepper motor while 640 images are recorded. This results in a three-dimensional array with $640\times 480\text{pixels}$ in the spatial dimension and 100 spectral channels with 5-nm resolution obtained in 4.6 s. Image sizes up to $1920\times 1080\text{pixels}$ in the spatial dimension are possible, but require longer acquisition time. Data processing (from raw data to a false-color image) and scanning are performed in parallel, and therefore, no additional time is needed for computation. Furthermore, real-time visualization of the processed false-color image is available during scanning. The field of view (FOV) and the spatial resolution are dependent on the distance to the object and the focal length of the OL. The FOV was measured for 5- and 10-cm distances, and this yielded $55\times 48\mathrm{m}{\mathrm{m}}^2$ and $100\times 83\mathrm{m}{\mathrm{m}}^2$, respectively.

Fig. 1

Schematic description of the laparoscopic hyperspectral camera system. The light of an external light source is coupled into the laparoscope for object illumination. An OL adapter focuses on different working distances and connects the laparoscope with the camera housing. The incident light of the visible spectral range is partially reflected from a BS on a rgbS. The remaining part of the visible light and all near-infrared light passes the BS and the high-pass FG toward the entrance slit of the spectrograph. The spectrograph consists of two achromatic lenses (L1 and L2), a transmission grating, and a monochromatic camera sensor (bwS). A stepper motor is moving the spectrograph during HSI (push-broom scanner).

Even though a color image from the rgbS and a false-color image calculated from spectral data show different areas of the object, the arrangement of both systems inside the camera is fixed. This allows the registration of both images after a one-time calculation of the homography. For this purpose, 25 corresponding points are manually annotated in both images. These points are the input for the OpenCV function findHomography¹⁹ that uses RANSAC²⁰ to calculate the homography ($3\times 3$ transformation matrix) between two planes by minimizing the back-projection error. Once the homography is estimated, the images are registered according to

Eq. (1)

$$D(x,y)=S(\frac{H_{11}x+H_{12}y+H_{13}}{H_{31}x+H_{32}y+H_{33}},\frac{H_{21}x+H_{22}y+H_{23}}{H_{31}x+H_{32}y+H_{33}}),$$

where $D(x,y)$ are the new pixel coordinates of the false-color image, $S(x,y)$ are the old coordinates, and $H$ is the homography.

2.3.

Reference Objects and Human Tissue Resectate

For white and color references the ColorChecker (X-Rite, Grand Rapids, Michigan), white paper and a tissue phantom with scattering properties similar to human tissue were used. The latter consists of silicone and barium sulfate in a ratio of 30:7 without additional absorbers. Figure 2 shows the laboratory set up with the laparoscope, light source, and white references. Different realistic object distances were evaluated for the laparoscopic HSI system at 5- to 10-cm distances in 1-cm steps. The spatial resolution was evaluated with the 1951 United States Air Force (USAF) resolution test chart at 5- and 10-cm object distances by calculating the Michelson contrast of the resolution elements at 500 and 800 nm. A sine curve was fitted to the intensity profile and the maximum and minimum intensities for the determination of the Michelson contrast were obtained. The used Michelson contrast cut off was 20%. Furthermore, measurements were performed during abdomino-thoracic esophageal resection in a patient with Barrett’s Esophagus (Siewert classification: adenocarcinoma of esophagogastric junction type I-II, neoadjuvant chemotherapy). Therefore, the resectate was placed in the laparoscopic trainer Laparo Advance (Laparo, Wroclaw, Poland) and the laparoscope was attached to a holding arm during HSI. The ambient light was switched off during all measurements to avoid spectral artifacts.

Fig. 2

Setup for measurements with the (1) laparoscopic hyperspectral camera, (2) rigid laparoscope, (3) custom near-infrared LED light source, (4) white tissue phantom, and (5) ColorChecker. Working distance ($\text{d}$) is controlled manually by moving the mount of (4) and (5) along the guide rail.

3.

Results

Standardized intensities of the three different light sources and white references are shown in Fig. 3(a). Ripples in the measured intensities above 800 nm are due to the spectral efficiency of the sensor and corrected during white balancing. The spatial intensity decrease of the light sources was evaluated during the same measurement from the illumination spot center to the border of the image. An intensity drop about 50% at 150 pixels distance from the spot center was observed for the LED NIR (575 nm), xenon and halogen (685 nm) light source [Fig. 3(b)]. The mean SNR of all sensor lines was 30 to 43 dB in the spectral range from 500 to 950 nm as illustrated in Figs. 3(c) and 3(d). Due to the intensity decrease at the image borders, up to 7 dB difference in the spatial dimension was observed. The TIVITA system with halogen lighting unit (50-cm object distance) showed an overall higher SNR compared to the laparoscopic system with LED or xenon light source. However, in the spectral range with high absorption coefficients of hemoglobin ($<600\mathrm{nm}$), the LED NIR light source provided a higher SNR.

Fig. 3

Spectral characteristic of the laparoscopic HSI system for different light sources (5-cm object distance): (a) standardized intensity of the LED NIR, xenon, and halogen light sources for three different white references (dotted lines represent the ColorChecker and dashed lines represent the white phantom). (b) Intensity decrease from illumination spot center to the border of the image for LED NIR (575 nm), xenon and halogen (685 nm). (c) SNR of the laparoscopic HSI system with LED NIR light source compared to the TIVITA system. The mean SNR of sensor lines 0 to 10, 240 to 250, and all lines in the spatial dimension are plotted for both systems. (d) Mean SNR of all sensor lines for the laparoscopic HSI (LED NIR and xenon light source) compared to the TIVITA system (halogen light).

The spectral distribution of the LED NIR light source measured at 5- to 10-cm distances in 1-cm steps is shown in Fig. 4. An intensity decrease proportional to ${1/\text{distance}}^{1.8}$ was observed, which approximately follows the inverse-square law for point sources.

Fig. 4

Intensity decrease in the image center measured with the laparoscopic HSI system and LED NIR light source at different object distances: (a) the measured intensities are approximately following the inverse-square law ($\text{intensity}\propto 1/{\text{distance}}^2$). (b) Logarithmic scale and normalized intensities indicate that the shape of the spectra is distance-independent.

At a 5-cm object distance and 500 nm, the spatial resolution was $320\mu \mathrm{m}$ in the $x$ dimension and $160\mu \mathrm{m}$ in the $y$ dimension. The spatial resolution approximately halved ($630\mu \mathrm{m}$ in the $x$ dimension and $315\mu \mathrm{m}$ in the $y$ dimension) for a doubled object distance (10 cm). A decreased spatial resolution was observed in the near-infrared range (800 nm) for the $y$ dimension ($500\mu \mathrm{m}$) though it remained constant for the $x$ dimension (scanning direction).

In Fig. 5, a static image of the color video from the color image sensor (rgbS), a color image calculated from HSI data of the bwS, and the registration of both images with semitransparent overlay are shown. The average root-mean-square error (RMSE) of the reflectance (500 to 900 nm) from 11 ColorChecker patches measured with the laparoscopic and TIVITA system was 0.03 ($\pm 0.02$). Three exemplary reflectance curves of these patches are given in Fig. 5(d) for both systems.

Fig. 5

Simultaneous video and HSI of the ColorChecker target: (a) video from the color image sensor (rgbS). (b) Registration and semitransparent overlay of the rgbS video and color image from HSI after one-time calculation of the homography. (c) Color image calculated during hyperspectral data acquisition. Pseudocolors are calculated from the measured reflectance between 530 and 725 nm, due to the lower SNR of the TIVITA system below this range. (d) Reflectance of three exemplary ColorChecker patches measured with the laparoscopic HSI and TIVITA system. Error bars indicating the standard deviations for the patches recorded with the laparoscopic system. Standard deviations for the TIVITA system are $<2\%$ and omitted for clarity.

The laparoscopic HSI system was successfully used in clinical set up for measurements of a resected adenocarcinoma of the esophagogastric junction from a patient with Barrett’s Esophagus. Data were acquired from extra- and intraluminal immediately after abdomino-thoracic esophageal resection. The corresponding structures in the color images acquired with the TIVITA and laparoscopic HSI system were matched postoperatively. Figure 6 shows the color images and OHI false-color images from the TIVITA and the laparoscopic HSI system. In addition, the mean absorbance spectra inside a corresponding ROI are illustrated. The data are plotted from 500 to 960 nm for the laparoscopic system and 550 to 1000 nm for TIVITA because of the low intensities outside this range. The RMSE of the two mean absorbance spectra inside the region of interest (ROI) is 0.02 (550 to 960 nm). For eight randomly selected ROIs, the average RMSE is 0.09 ($\pm 0.05$).

Fig. 6

Intraluminal HSI of the resected adenocarcinoma of the esophagogastric junction: color image and OHI calculated from spectral data acquired with (a), (b) TIVITA and (c), (d) laparoscopic HSI system. The corresponding FOV is labeled as a blue circle. (e) Mean absorbance spectrum of the pixels inside the black square for both HSI systems. (f) Mean absorbance spectrum of adenocarcinoma (pixels inside yellow circle), esophageal and gastric mucosa (outside of the FOV) measured with the laparoscopic HSI system and labeled by a surgeon according to visual and haptic perception.

4.

Discussion

The intraoperative estimation of tissue perfusion and differentiation is essential during most laparoscopic procedures. HSI can support visual decision making, but is not yet ready for routine laparoscopic use. An HSI system with simultaneous high-resolution video for minimal-invasive surgery may reduce the risk of perfusion related complications and assist the identification of lesions and risk structures.

In this work, a laparoscopic system capable of spectral imaging from 500 to 1000 nm with high spatial and spectral resolution in 4.6 s was described. The raw spectral data are processed and visualized in real time during acquisition, hence no additional time is needed. Furthermore, the handheld device provides high-resolution video during HSI and allows pushbroom imaging without the previously reported limitations of moving the object or the laparoscope. The registration of false-color images derived from the hyperspectral data (e.g., perfusion maps) and high-resolution video provides augmented visual information. The measured ColorChecker reflectance data were consistent with a commercially available medical HSI device for the practical object distance of 10 cm. Moreover, corresponding regions of the resected esophagogastric junction showed the same spectral signature for both systems although the light source and measurement distance were different.

Additionally, OHI false-color images of the resected tissue from the open and laparoscopic HSI system showed the same qualitative results. Quantitative deviations of OHI were observed and could be due to the time delay between measurements, repositioning in the laparoscopic trainer, and glare artifacts, which needs to be investigated more closely.

Only minor differences were observed for the spectral shape of the three different white references, however, variations in absolute intensity were present and must be considered when comparing reflectance values. Even though the used light sources slightly differed in their spatial distribution, all showed a strong intensity decline in the FOV. A homogenous illumination would improve the SNR at the image borders.

The entrance slit can be moved automatically by the integrated motor to scan the image or can be stopped at any position if desired. This design also allows manual scanning, such as that recently described by Yoon et al.¹⁶ for flexible endoscopy.

Currently, the SNR in the near-infrared range is limited by the available light sources and the transmission of the used laparoscopes. Work is in progress to solve this problem. The size of the system is already comparable to usual laparoscopic cameras. However, the housing could become even smaller by removing parts of the processing hardware. To avoid motion artifacts, the laparoscope was attached to a holding arm during scanning. Movements of the tissue, for instance due to breathing, cannot be prevented and thus should be aimed at further reduction of HSI acquisition time.

In the authors’ opinion, rigid laparoscopic HSI has great potential for fast and repeatable quantification of tissue perfusion as reported by Clancy et al.²¹ Although the measured reflectance data agreed with the results from the TIVITA system, studies on perfused tissue need to verify that the approved perfusion indices are valid for laparoscopic use.

Laparoscopic HSI in the visible and near-infrared range with short processing time and simultaneous video is feasible and fulfills the requirements for routine clinical use. Studies with more clinical data are needed to evaluate the benefits of laparoscopic HSI for intraoperative applications.

5.

Conclusions

An HSI system with high spatial and spectral resolution in the visible and near-infrared range and simultaneous color video for minimally invasive surgery was presented. Reference objects and resected tissue were used to show the consistency of the measured data from the laparoscopic HSI and the approved TIVITA system. The compact handheld design, rapid HSI, and high-resolution video at the same time enable high clinical practicability. This work may support the translation of HSI to routine laparoscopic surgery. Future studies on the added value of laparoscopic HSI for the patient outcome are needed.