Background and objective

Modern imaging scanners produce an ever-growing body of 3D/4D multimodal data requiring image analytics and visualization of fused images, segmentations, and information. For the latter, augmented reality (AR) with head-mounted displays (HMDs) has shown potential. This work describes a framework (FI3D) for interactive immersion with data, integration of image processing and analytics, and rendering and fusion with an AR interface.

Methods

The FI3D was designed and endowed with modules to communicate with peripherals, including imaging scanners and HMDs, and to provide computational power for data acquisition and processing. The core of FI3D is deployed to a dedicated computational unit that performs the computationally demanding processes in real-time, and the HMD is used as a display output peripheral and an input peripheral through gestures and voice commands. FI3D offers user-made processing and analysis dedicated modules. Users can customize and optimize these for a particular workflow while incorporating current or future libraries.

Results

The FI3D framework was used to develop a workflow for processing, rendering, and visualization of CINE MRI cardiac sets. In this version, the data were loaded from a remote database, and the endocardium and epicardium of the left ventricle (LV) were segmented using a machine learning model and transmitted to a HoloLens HMD to be visualized in 4D. Performance results show that the system is capable of maintaining an image stream of one image per second with a resolution of 512 × 512. Also, it can modify visual properties of the holograms at 1 update per 16 milliseconds (62.5 Hz) while providing enough resources for the segmentation and surface reconstruction tasks without hindering the HMD.

Conclusions

We provide a system design and framework to be used as a foundation for medical applications that benefit from AR visualization, removing several technical challenges from the developmental pipeline.

中文翻译：

---

模块化且可扩展的计算框架，用于通过全息增强现实界面交互式沉浸到成像数据中

背景和目标

现代成像扫描仪可生成不断增长的3D / 4D多模态数据，需要图像分析以及融合图像，分割和信息的可视化。对于后者，具有头戴式显示器（HMD）的增强现实（AR）已显示出潜力。这项工作描述了一个框架（FI3D），用于与数据进行交互沉浸，图像处理和分析的集成以及与AR接口的渲染和融合。

方法

FI3D的设计并赋予了与外围设备（包括成像扫描仪和HMD）进行通信的模块，并为数据采集和处理提供了计算能力。FI3D的核心被部署到一个专用的计算单元，该单元实时执行计算需求的过程，并且HMD通过手势和语音命令用作显示输出外围设备和输入外围设备。FI3D提供用户定制的处理和分析专用模块。用户可以在合并当前或将来的库的同时针对特定的工作流程自定义和优化这些库。

结果

FI3D框架用于开发工作流程，以处理，渲染和可视化CINE MRI心脏装置。在此版本中，数据是从远程数据库中加载的，并且使用机器学习模型对左心室（LV）的心内膜和心外膜进行了分割，并传输到HoloLens HMD以4D可视化。性能结果表明，该系统能够以512×512的分辨率维持每秒一幅图像的图像流。此外，它还可以在提供足够资源的情况下以每16毫秒（62.5 Hz）更新1次的方式修改全息图的视觉特性。用于分割和曲面重建任务，而不会阻碍HMD。

结论

我们提供了系统设计和框架，可作为受益于AR可视化的医疗应用的基础，从而消除了开发流程中的一些技术难题。