We address the problem of offline handwritten diagram recognition. Recently, it has been shown that diagram symbols can be directly recognized with deep learning object detectors. However, object detectors are not able to recognize the diagram structure. We propose Arrow R-CNN, the first deep learning system for joint symbol and structure recognition in handwritten diagrams. Arrow R-CNN extends the Faster R-CNN object detector with an arrow head and tail keypoint predictor and a diagram-aware postprocessing method. We propose a network architecture and data augmentation methods targeted at small diagram datasets. Our diagram-aware postprocessing method addresses the insufficiencies of standard Faster R-CNN postprocessing. It reconstructs a diagram from a set of symbol detections and arrow keypoints. Arrow R-CNN improves state-of-the-art substantially: on a scanned flowchart dataset, we increase the rate of recognized diagrams from 37.7 to 78.6%.

我们解决了离线手写图识别的问题。近来，已经表明，可以通过深度学习对象检测器直接识别图符号。但是，对象检测器无法识别图结构。我们提出了Arrow R-CNN，这是第一个用于手写图中的联合符号和结构识别的深度学习系统。Arrow R-CNN扩展了Faster R-CNN对象检测器，它具有箭头头和尾部的关键点预测器以及可识别图的后处理方法。我们提出了针对小型图数据集的网络体系结构和数据扩充方法。我们的图形感知后处理方法解决了标准Faster R-CNN后处理的不足。它从一组符号检测和箭头关键点重建图。