We propose a three-dimensional (3D) multimodal medical imaging system that combines freehand ultrasound and structured light 3D reconstruction in a single coordinate system without requiring registration. To the best of our knowledge, these techniques have not been combined as a multimodal imaging technique. The system complements the internal 3D information acquired with ultrasound with the external surface measured with the structured light technique. Moreover, the ultrasound probe’s optical tracking for pose estimation was implemented based on a convolutional neural network. Experimental results show the system’s high accuracy and reproducibility, as well as its potential for preoperative and intraoperative applications. The experimental multimodal error, or the distance from two surfaces obtained with different modalities, was 0.12 mm. The code is available in a Github repository.

中文翻译：

---

基于徒手超声和结构光的三维多模态医学成像系统

我们提出了一种三维（3D）多模态医学成像系统，该系统将徒手超声和结构光3D重建结合在一个坐标系中，而无需注册。据我们所知，这些技术还没有被组合成一种多峰成像技术。该系统将利用超声获取的内部3D信息与通过结构化光技术测量的外表面进行补充。此外，基于卷积神经网络实现了用于姿态估计的超声探头的光学跟踪。实验结果表明该系统具有很高的准确性和可重复性，并且具有术前和术中应用的潜力。实验的多峰误差或到以不同模态获得的两个表面之间的距离为0.12 mm。