Neural joint attention code search over structure embeddings for software Q&A sites

Highlights

• Propose a joint attention framework for code search in software Q&A sites.

• Introduce structure embeddings to enhance searching for code fragments.

• Achieve encouraging performance on four different programming languages.

• Construct high-quality evaluation corpus for code search in software Q&A sites.

Abstract

Code search is frequently needed in software Q&A sites for software development. Over the years, various code search engines and techniques have been explored to support user query. Early approaches often utilize text retrieval models to match textual code fragments for natural query, but fail to build sufficient semantic correlations. Some recent advanced neural methods focus on restructuring bi-modal networks to measure the semantic similarity. However, they ignore potential structure information of source codes and the joint attention information from natural queries. In addition, they mostly focus on specific code structures, rather than general code fragments in software Q&A sites.

In this paper, we propose NJACS, a novel two-way attention-based neural network for retrieving code fragments in software Q&A sites, which aligns and focuses the more structure informative parts of source codes to natural query. Instead of directly learning bi-modal unified vector representations, NJACS first embeds the queries and codes using a bidirectional LSTM with pre-trained structure embeddings separately, then learns an aligned joint attention matrix for query-code mappings, and finally derives the pooling-based projection vectors in different directions to guide the attention-based representations. On different benchmark search codebase collected from StackOverflow, NJACS outperforms state-of-art baselines with 7.5% to 6% higher Recall@1 and MRR, respectively. Moreover, our designed structure embeddings can be leveraged for other deep-learning-based software tasks.

Introduction

In software development activities, source code examples are critical for concept understanding, applying fixes, extending software functionalities, etc. Code search increases developers’ productivity and reduces duplication of effort. The goal of code search is to retrieve code fragments from a large code corpus that most closely match a developer’s intent, which is expressed in natural language. Previous studies have revealed that more than 60% of developers search code in Q&A sites every day (Hoffmann et al., 2007). As online software Q&A sites (e.g., StackOverflow, 0000, Github, 0000, Krugle, 0000) contain millions of open source projects with code fragments, many search engines are designed to retrieve code solutions for natural language (NL) queries issued by developers. Unfortunately, these search engines often return with unrelated or other language codes in search results (Sadowski et al., 2015), even when reformulated queries are provided (Carpineto and Romano, 2012).

Recent work from both academia and industry has enabled more advanced code search using different techniques. Consider the examples in Fig. 1, we collect three different NL queries and their code solutions from Post#133031 (StackOverflow, 2009a), Post#19370921 (StackOverflow, 2012b) and Post#11621614 (StackOverflow, 2012a) in StackOverflow. There are still several challenges for previous work as follow:

(1) Common features mismatch using IR-based techniques. Previous methods for code search applied with information retrieval (IR) techniques, but most of them depend greatly on the quality of matching terms contained in both NL queries and source codes (Haiduc et al., 2013). As NL query and source code in the $〈Q_1,C_1〉$ (Shown in Fig. 1) pair is heterogeneous, they may not share enough common lexical tokens, synonyms, or language structures, especially in short queries. Although most approaches provide effective ways for query reformulation (Paik et al., 2014, Lu et al., 2018, Sirres et al., 2018) (e.g., query expansion, text paraphrase), over specific queries return no effective results (Grechanik et al., 2010). Moreover, these extractive methods cannot handle irrelevant/noisy keywords in NL queries effectively. In fact, most NL queries and source codes may only be semantically related, as similar in cross-language text retrieval or machine translation. Thus, deep-learning-based methods (Iyer et al., 2016, Huo et al., 2016, Chen and Zhou, 2018) are further proposed to mine the semantic relevance between NL query and source code. Similarly, we designed NJACS for code search in software Q&A sites using a two-way attention-based neural network, which can perform NL semantic query for various programming languages.

(2) Lack of integral code representation via structure embeddings. More recently, various neural models are applied to learning the unified representation of both source code and NL query, such as code annotation (Huo et al., 2016), bug localization (Chen and Zhou, 2018) and software repair (White et al., 2016), etc. Similarly with code search, Gu et al. (2018) propose a bi-modal neural network named CODEnn, but limit its application to Java code fragments. CODEnn relies on extracting subelements (including method names, tokens, and API sequences Gu et al., 2016), thus cannot generate the overall semantic representation of the code structure. Unfortunately, like the $〈Q_2,C_2〉$ (Shown in Fig. 1) pair belongs to SQL domain with no method types, these code-split-based embedding methods (Paik et al., 2014, Grechanik et al., 2010) may not be suitable for code fragments of other structures and program types. Thus, compared to CODEnn, Sachdev et al., 2018, Yao et al., 2019 and Cambronero et al. (2019) present the different supervised neural models, labeled NCS, CoaCor and UNIF respectively that successfully learned code embeddings and integral code representation using corpora of query-code mappings. Nevertheless, these methods use one-hot representation (Turian et al., 2010), pre-trained word2vec embeddings (Mikolov et al., 2013a) or randomly initialized word embeddings (Gu et al., 2018, Yao et al., 2019) for pre-training neural networks, which lack of introducing external pre-training information to improve model performance similar to BERT (Devlin et al., 2018) used in NL processing (NLP). Unlike them, NJACS introduces pre-trained structure embeddings to further capture more structure information to enhance the integral code semantic representation.

(3) Lack of modeling attention focus and structure information: Most existing neural approaches (Huo et al., 2016, Chen and Zhou, 2018, Gu et al., 2018, Yao et al., 2019, Cambronero et al., 2019, Hu et al., 2018) utilize RNNs (Hochreiter and Schmidhuber, 1997) (Recurrent Neural Networks), CNNs (Pattanayak, 2017) (Convolutional Neural Networks) or DQNs (Mnih et al., 2013) (Deep Q-learning Networks) based bi-modal network for code search. They generally ignore the joint attention information between NL query and source code, thus cannot effectively capture deeper semantic matching signals. As for the $〈Q_3,C_3〉$ (Shown in Fig. 1) pair, understanding the focus of NL queries (e.g., important terms “extract”, “regex” and “text”) is helpful for retrieving more relevant codes that talk about “text extraction” problem. However, such methods do not explicitly model query focus (attention links are shown in Fig. 1). Recent studies (Iyer et al., 2016, Devlin et al., 2018, Wan et al., 2018) have also proved that attention mechanism can be successfully applied in code summarization (Iyer et al., 2016, Wan et al., 2018) and various NLP tasks like Machine Reading Comprehension (MRC) (Devlin et al., 2018). Besides, many other approaches (Huo et al., 2016, Gu et al., 2018, Sachdev et al., 2018, Cambronero et al., 2019) fail to consider the potential structure information of source code, which carries additional semantics to the program functionality besides the lexical terms. For NJACS, its two-way attention mechanism can capture sufficiently matching terms to align and focus the more structural informative parts of source code to NL query.

To address aforementioned issues, we propose a novel neural network named NJACS for code search in software Q&A sites, which leverages two attention mechanisms as global attention and attentive pooling. To solve the problem of common features mismatch, NJACS learns an enhanced joint representation of NL query and source code, which captures the semantics in lexicon along with code structure and introduces focus information between NL query and code fragment. In NJACS, the desired codes can be retrieved from most common queries without frequent reformulation.

To the best of our knowledge, we are the first to propose joint-attention-based code search. The main contributions of our work are as follows.

• We propose NJACS, a neural joint attention network that utilizes two-way attention mechanism to improve unified representation learning from both natural language and programming languages for retrieving relevant code solutions in software Q&A sites.

• We design particular code structure embeddings for pre-training operations with respect to different programming languages and code structures, which is able to capture semantics of the program from both lexical and program structural perspectives.

• We construct large-scale software repositories containing almost 10 million pairs and 200 sampled examples with most candidate codes, all extracted from use-case scenarios like StackOverflow, and conduct comprehensive experiments on the repositories.

The rest of this paper is organized as follows. The next section gives the definitions and motivation. Sections 3 Learning structure embeddings, 4 NJACS: Neural joint attention code search using structure embeddings describes the detailed design of the structure embeddings and the NJACS model, respectively. Experiments are shown in Section 5. Sections 6 Evaluation and results, 7 Discussions gives the evaluation results and discussion. Section 8 presents the related work. Finally, we conclude the paper in Section 9.