Towards automatically generating block comments for code snippets

Abstract

Code commenting is a common programming practice of practical importance to help developers review and comprehend source code. There are two main types of code comments for a method: header comments that summarize the method functionality located before a method, and block comments that describe the functionality of the code snippets within a method. Inspired by the effectiveness of deep learning techniques in the NLP field, many studies focus on using the machine translation model to automatically generate comment for the source code. Because the data set of block comments is difficult to collect, current studies focus more on the automatic generation of header comments than that of block comments. However, block comments are important for program comprehension due to their explanation role for the code snippets in a method. To fill the gap, we have proposed an approach that combines heuristic rules and learning-based method to collect a large number of comment-code pairs from 1,032 open source projects in our previous study. In this paper, we propose a reinforcement learning-based method, RL-BlockCom, to automatically generate block comments for code snippets based on the collected comment-code pairs. Specifically, we utilize the abstract syntax tree (i.e., AST) of a code snippet to generate a token sequence with a statement-based traversal way. Then we propose a composite learning model, which combines the actor-critic algorithm of reinforcement learning with the encoder-decoder algorithm, to generate block comments. On the data set of the comment-code pairs, the BLEU-4 score of our method is 24.28, which outperforms the baselines and state-of-the-art in comment generation.

Introduction

Code commenting, an integral part of software development [1], has been a standard practice in the industry [2], [3]. Comments improve software maintainability through helping developers understand the source code [4], [5]. Obviously, high-quality code comments play an important role in most software activities, such as: code review [6], [7], [8] and program comprehension [9], [10].

Owing to the importance of code comments, a number of approaches [11], [12], [13], [14] have been proposed to automatically generate comments via “translating” the programming language source code to natural language comment benefitting from the effectiveness of deep learning techniques. There are two types of comments used for annotating a method, i.e., header comments and block comments [15], [16]. Header comments are used to summarize the method functionality that locate before a method (e.g., the comment at line 3 in Code Example 1), while block comments are used to describe the functionality of a code snippet in a method (e.g., the comment at lines 9 and 10 in Code Example 1). The scope of a header comment is the whole method, while the scope of a block comment is variable, i.e., from one code line to several continuous code lines.

Due to the variability of the scope of a block comment, it is difficult to automatically collect a data set for block comments that is used to train a block comment generation model. Meanwhile, a considerable amount of work will be involved if using a manual way (i.e., manually label data) to collect the data set. As a result, current studies mainly focus on the automatic generation of header comment [11], [12], [13], [14] (because the pair of header comment and the method is easy to collect). However, block comments are as important as header comments in code understanding [17], [18]. In most time, header comments summarize the functionality of a method at a higher level, while block comments focus on describing the detailed implementation of the code, and the two complement each other [17].

Therefore, it is necessary to build a data set for block comments and further to train a model on the data set to automatically generate block comments. Recently, some studies [11], [12], [13] have adopted deep learning approaches to generate header comments by combining the machine translation model and the structural and semantic information within the Java methods. Although deep learning techniques are successful in the first step toward automatic header comment generation, it is difficult to directly apply these models to the automatic generation of block comments.

Intuitively, the source code covered by the block comments is more fragmented comparing with that covered by the header comments, i.e., a code snippet vs. a complete method. A method is a unit with complete semantics and structure, while a code snippet lacks the context information when we extract them from the source code to form a comment-code pair. As a result, it is more difficult to employ a general machine translation algorithm to generate the comment on a “fragmented” data set, because the mapping space between the code statements and comment words becomes very huge.

In this paper, to address the challenge of collecting the data set of the block comments, we employ the approach proposed in our previous work [19] that combines heuristic rules and learning-based method to identify the scope of a block comment. The heuristic rules can segment the source code into initial comment-code pairs, which can reduce the manual validation effort. Then our approach extracts three dimensions of features including comment features, code features and comment-code relationship features to further identify the scope of a block comment. After that, we apply this approach to automatically collect a data set that contains more than 123,900 comment-code pairs (after the filtering) from 1,032 open-source Java projects. Our approach achieves an accuracy of 81.15% in detecting the scope of block comments.

To address the challenge of automatically generating block comments, we propose a composite learning model, RL-BlockCom, which combines the actor-critic algorithm of reinforcement learning with the encoder-decoder algorithm, to generate block comments. The combination of the two algorithms can resolve the loss-evaluation mismatch issue. More importantly, the reinforcement learning algorithm can learn effective strategies from infinite mapping space to decide the profit-maximizing actions, which is ideal for learning the mappings between the comment words and the fragmented code. Specifically, RL-BlockCom utilizes the abstract syntax tree of a code snippet to generate a token sequence with a statement-based traversal way, then we train RL-BlockCom on the block comment data set. In the evaluation, the BLEU-4 score of RL-BlockCom is 24.28, which outperforms the baseline and the state-of-the-art methods. To facilitate research and application, our RL-BlockCom is available at https://github.com/huangshh/RLComGen.

The contributions of our work are as follows:

• We propose a statement-based AST traversal algorithm to generate the code token sequence with preserving the semantic, syntactic and structural information in the code snippet.

• We propose a composite approach named RL-BlockCom that generates block comments with combining the actor-critic algorithm of reinforcement learning and the encoder-decoder algorithm.

• We implement RL-BlockCom and perform careful experiments with the collected block comment data set to evaluate its performance. The experimental results show that RL-BlockCom outperforms the baselines and state-of-the-art in comment generation.

The rest of this paper is organized as follows. Section 2 shows the overall framework of our approach. Section 3 presents the data collection and the main method of learning to identify the scope of block comments. The detailed steps of learning to generate block comments are discussed in Section 4. Section 5 presents the experiment setting, and Section 6 shows the experiment results. More discussions are presented in Section 7, while Section 8 discusses related works. Section 9 outlines the threats to validity and Section 10 summarizes our approach and outlines directions of future work.