A real-time motion capture system is presented which uses input from multiple standard video cameras and inertial measurement units (IMUs). The system is able to track multiple people simultaneously and requires no optical markers, specialized infra-red cameras or foreground/background segmentation, making it applicable to general indoor and outdoor scenarios with dynamic backgrounds and lighting. To overcome limitations of prior video or IMU-only approaches, we propose to use flexible combinations of multiple-view, calibrated video and IMU input along with a pose prior in an online optimization-based framework, which allows the full 6-DoF motion to be recovered including axial rotation of limbs and drift-free global position. A method for sorting and assigning raw input 2D keypoint detections into corresponding subjects is presented which facilitates multi-person tracking and rejection of any bystanders in the scene. The approach is evaluated on data from several indoor and outdoor capture environments with one or more subjects and the trade-off between input sparsity and tracking performance is discussed. State-of-the-art pose estimation performance is obtained on the Total Capture (mutli-view video and IMU) and Human 3.6M (multi-view video) datasets. Finally, a live demonstrator for the approach is presented showing real-time capture, solving and character animation using a light-weight, commodity hardware setup.

中文翻译：

---

来自多视图视频和 IMU 的实时多人运动捕捉

提出了一种实时运动捕捉系统，该系统使用来自多个标准摄像机和惯性测量单元 (IMU) 的输入。该系统能够同时跟踪多人，不需要光学标记、专门的红外摄像头或前景/背景分割，使其适用于具有动态背景和照明的一般室内和室外场景。为了克服先前视频或仅 IMU 方法的局限性，我们建议在基于在线优化的框架中使用多视图、校准视频和 IMU 输入的灵活组合以及位姿先验，这允许完整的 6-DoF 运动恢复包括四肢的轴向旋转和无漂移的全局位置。提出了一种将原始输入 2D 关键点检测分类并分配到相应主题的方法，该方法有助于多人跟踪和拒绝场景中的任何旁观者。该方法根据来自具有一个或多个对象的多个室内和室外捕获环境的数据进行评估，并讨论了输入稀疏性和跟踪性能之间的权衡。在 Total Capture（多视图视频和 IMU）和 Human 3.6M（多视图视频）数据集上获得了最先进的姿态估计性能。最后，展示了该方法的现场演示器，展示了使用轻量级商品硬件设置的实时捕获、求解和角色动画。该方法根据来自具有一个或多个对象的多个室内和室外捕获环境的数据进行评估，并讨论了输入稀疏性和跟踪性能之间的权衡。在 Total Capture（多视图视频和 IMU）和 Human 3.6M（多视图视频）数据集上获得了最先进的姿态估计性能。最后，展示了该方法的现场演示器，展示了使用轻量级商品硬件设置的实时捕获、求解和角色动画。该方法根据来自具有一个或多个对象的多个室内和室外捕获环境的数据进行评估，并讨论了输入稀疏性和跟踪性能之间的权衡。在 Total Capture（多视图视频和 IMU）和 Human 3.6M（多视图视频）数据集上获得了最先进的姿态估计性能。最后，展示了该方法的现场演示器，展示了使用轻量级商品硬件设置的实时捕获、求解和角色动画。