There is a growing interest in leveraging Deep Learning (DL) for automating Software Engineering tasks such as program completion. In this paper, we leverage Recurrent Neural Networks (RNNs) for Abstract Syntax Tree (AST)-based code completion. Our approach converts source code into AST nodes and a language model predicts the type and value attributes of next tokens. Our work demonstrates that the attention augmented RNN-based language models are able to understand local context and copy from recent past tokens which have never appeared in the training data set. We observed a drop of performances of both type and value predictions when using a traditional pointer network architecture for out-of-vocabulary (OoV) copying and context understanding, which we call multi-task conflict. To address this challenge, we have devised a new structure of self-attention called Split Attention, where two separate dot-product layers are applied to different parts of the history cache. Based on this structure, we propose a new network called Split Attention Pointer Network (SAPN), which is efficient and flexible in both learning local context and copying OoV tokens from history. The empirical results suggest that our model is superior in syntax-aware generation and OoV token prediction by demonstrating attention behavior similar to human programmers. The results also indicate that our model out performs previous state-of-the-art approaches by more than 6% on widely recognized program completion benchmarks.

中文翻译：

---

用于源代码语言建模的拆分注意力指针网络

人们对利用深度学习 (DL) 来自动化软件工程任务（例如程序完成）的兴趣与日俱增。在本文中，我们利用循环神经网络 (RNN) 进行基于抽象语法树 (AST) 的代码完成。我们的方法将源代码转换为 AST 节点，语言模型预测下一个标记的类型和值属性。我们的工作表明，基于注意力增强的 RNN 语言模型能够理解本地上下文并从最近从未出现在训练数据集中的过去标记中复制。当使用传统指针网络架构进行词汇外 (OoV) 复制和上下文理解时，我们观察到类型和值预测的性能下降，我们称之为多任务冲突。为了应对这一挑战，我们设计了一种新的自我注意结构，称为拆分注意，其中两个独立的点积层应用于历史缓存的不同部分。基于这种结构，我们提出了一种称为拆分注意力指针网络（SAPN）的新网络，该网络在学习本地上下文和从历史中复制 OoV 令牌方面都高效且灵活。实证结果表明，我们的模型通过展示类似于人类程序员的注意力行为，在语法感知生成和 OoV 令牌预测方面具有优势。结果还表明，我们的模型在广泛认可的程序完成基准上比以前最先进的方法高出 6% 以上。我们提出了一个称为拆分注意力指针网络 (SAPN) 的新网络，该网络在学习本地上下文和从历史中复制 OoV 令牌方面都高效且灵活。实证结果表明，我们的模型通过展示类似于人类程序员的注意力行为，在语法感知生成和 OoV 令牌预测方面具有优势。结果还表明，我们的模型在广泛认可的程序完成基准上比以前最先进的方法高出 6% 以上。我们提出了一个称为拆分注意力指针网络 (SAPN) 的新网络，该网络在学习本地上下文和从历史中复制 OoV 令牌方面都高效且灵活。实证结果表明，我们的模型通过展示类似于人类程序员的注意力行为，在语法感知生成和 OoV 令牌预测方面具有优势。结果还表明，我们的模型在广泛认可的程序完成基准上比以前最先进的方法高出 6% 以上。