We propose a novel approach for predicting on-going action with the assistance of a low-cost depth camera. Our approach introduces a soft regression-based early prediction framework. In this framework, we estimate soft labels for the subsequences at different progress levels, jointly learned with an action predictor. Our formulation of soft regression framework 1) overcomes a usual assumption in existing early action prediction systems that the progress level of on-going sequence is given in the testing stage; and 2) presents a theoretical framework to better resolve the ambiguity and uncertainty of subsequences at early performing stage. The proposed soft regression framework is further enhanced in order to take the relationships among subsequences and the discrepancy of soft labels over different classes into consideration, so that a Multiple Soft labels Recurrent Neural Network (MSRNN) is finally developed. For real-time performance, we also introduce a new RGB-D feature called “local accumulative frame feature (LAFF)”, which can be computed efficiently by constructing an integral feature map. Our experiments on three RGB-D benchmark datasets and an unconstrained RGB action set demonstrate that the proposed regression-based early action prediction model outperforms existing models significantly and also show that the early action prediction on RGB-D sequence is more accurate than that on RGB channel.

中文翻译：

---

通过软回归进行早期行动预测


我们提出了一种在低成本深度相机的帮助下预测正在进行的动作的新方法。我们的方法引入了基于软回归的早期预测框架。在此框架中，我们估计不同进度级别的子序列的软标签，并与动作预测器共同学习。我们制定的软回归框架1）克服了现有早期行动预测系统中的常见假设，即在测试阶段给出正在进行的序列的进度水平； 2）提出了一个理论框架，以更好地解决早期执行阶段子序列的模糊性和不确定性。进一步增强了所提出的软回归框架，以考虑子序列之间的关系以及不同类别的软标签的差异，从而最终开发出多软标签递归神经网络（MSRNN）。为了实现实时性能，我们还引入了一种新的 RGB-D 特征，称为“局部累积帧特征（LAFF）”，可以通过构造积分特征图来高效计算该特征。我们在三个 RGB-D 基准数据集和无约束 RGB 动作集上的实验表明，所提出的基于回归的早期动作预测模型显着优于现有模型，并且还表明 RGB-D 序列上的早期动作预测比 RGB 上的早期动作预测更准确渠道。