Digital painting is a process of creating a digital artwork using modern human-computer interaction technologies. One of the core enabling technologies is the real-time tracking of user’s strokes, which is generally supplied by a digital tablet with a stylus. While the digital tablet technology provides highly accurate tracking, the drawing should be done with a rigid stylus on a plastic surface. This sometimes destroys the realism of drawing, such as interaction with the digital tablet cannot provide the feedback of subtle texture, friction of the paper/fabric canvas and tension of soft painting brush. This becomes particularly problematic for traditional painting artists who are trained with and prefer real painting brush and paper/fabric canvas. Thus, the aim of this work is to present an alternative solution where the user’s strokes can be tracked even when the actual brush and canvas are used. To this end, we proposed two approaches for digitally tracking the tip of flexible bristles of a soft brush, so that the painting can be created digitally on a computer. The first approach captures the silhouette of deforming bristles using a pair of well-aligned infra-red (IR) cameras, which extracts the tip from the silhouette, and reconstructs the 2D position of the tip. The second approach predicts the brush tip position through a deep ensemble network-based approach where the relationship between the brush tip position and brush handle pose are trained with our novel model comprising of Long-Short Term Memory Autoencoder and 1-D Convolutional Neural Network. The trained model is used to predict the brush tip position in realtime. Both approaches extensively evaluated through multiple tests. Furthermore, our model outperforms the state-of-the-art models.

中文翻译：

---

在真实画布上跟踪柔性画笔笔尖：基于轮廓和基于深度集成网络的方法

数字绘画是使用现代人机交互技术创建数字艺术品的过程。核心使能技术之一是对用户笔画的实时跟踪，通常由带有手写笔的数字平板电脑提供。虽然数字平板技术提供了高度精确的跟踪，但绘图应使用塑料表面上的刚性手写笔完成。这有时会破坏绘画的真实感，例如与数位板的交互无法提供细微纹理、纸/织物画布的摩擦力和软画笔的张力等反馈。这对于受过训练并更喜欢真正的画笔和纸/织物帆布的传统绘画艺术家来说尤其成问题。因此，这项工作的目的是提出一种替代解决方案，即使在使用实际画笔和画布时，也可以跟踪用户的笔画。为此，我们提出了两种数字跟踪软刷柔性刷毛尖端的方法，以便可以在计算机上以数字方式创建绘画。第一种方法使用一对对齐良好的红外 (IR) 相机捕捉变形刷毛的轮廓，从轮廓中提取尖端，并重建尖端的 2D 位置。第二种方法通过基于深度集成网络的方法预测画笔尖端位置，其中画笔尖端位置和画笔手柄姿势之间的关系使用我们的新模型进行训练，该模型由长短期记忆自动编码器和一维卷积神经网络组成。训练好的模型用于实时预测笔尖位置。这两种方法都通过多次测试进行了广泛评估。此外，我们的模型优于最先进的模型。