Constructing and evaluating automated literature review systems

Abstract

Automated literature reviews have the potential to accelerate knowledge synthesis and provide new insights. However, a lack of labeled ground-truth data has made it difficult to develop and evaluate these methods. We propose a framework that uses the reference lists from existing review papers as labeled data, which can then be used to train supervised classifiers, allowing for experimentation and testing of models and features at a large scale. We demonstrate our framework by training classifiers using different combinations of citation- and text-based features on 500 review papers. We use the R-Precision scores for the task of reconstructing the review papers’ reference lists as a way to evaluate and compare methods. We also extend our method, generating a novel set of articles relevant to the fields of misinformation studies and science communication. We find that our method can identify many of the most relevant papers for a literature review from a large set of candidate papers, and that our framework allows for development and testing of models and features to incrementally improve the results. The models we build are able to identify relevant papers even when starting with a very small set of seed papers. We also find that the methods can be adapted to identify previously undiscovered articles that may be relevant to a given topic.

Appendix

Example of autoreview results

Below is a sample of results (random samples of true positives, false positives, true negatives, and false negatives) from the autoreview classifier using the references from Fortunato (2010)—a review on Community Detection in Graphs—with a random seed of 5. The “Rank” represents the position of the candidate paper when ordered descending by the classifier’s score. The false positives, while not in the original reference list, still seem to be relevant to the topic (e.g., “Overlapping Community Search for Social Networks”). The true negatives tend to have lower scores than the false negatives, suggesting that the assigned score does tend to predict relevant documents, even if they are below the cutoff.

True Positives
Rank	Title	Year
10	Role Models For Complex Networks	2007
21	Adaptive Clustering Algorithm For Community Detection In Complex Networks	2008
24	Random Field Ising Model And Community Structure In Complex Networks	2006
50	Bayesian Approach To Network Modularity	2008
58	The Effect Of Size Heterogeneity On Community Identification In Complex Networks	2006
76	Loops And Multiple Edges In Modularity Maximization Of Networks	2010
97	Synchronization Interfaces And Overlapping Communities In Complex Networks	2008
118	The Analysis And Dissimilarity Comparison Of Community Structure	2006
119	Searching For Communities In Bipartite Networks	2008
208	Epidemic Spreading In Scale-Free Networks	2001

False Positives
Rank	Title	Year
72	Modularity From Fluctuations In Random Graphs And Complex Networks	2004
94	Clustering Coefficient And Community Structure Of Bipartite Networks	2008
98	Detecting Overlapping Community Structures In Networks	2009
106	Size Reduction Of Complex Networks Preserving Modularity	2007
129	Extracting Weights From Edge Directions To Find Communities In Directed Networks	2010
146	Identifying The Role That Animals Play In Their Social Networks	2004
150	Seeding The Kernels In Graphs: Toward Multi-Resolution Community Analysis	2009
159	Overlapping Community Search For Social Networks	2010
162	Modularity Clustering Is Force-Directed Layout	2009
185	Cartography Of Complex Networks: Modules And Universal Roles	2005

True Negatives
Rank	Title	Year
2967	Graph Models Of Complex Information-Sources	1979
120959	Parallel Distributed Network Characteristics Of The Dsct	1992
322251	Hidden Semantic Concept Discovery In Region Based Image Retrieval	2004
327308	A Multilevel Matrix Decomposition Algorithm For Analyzing Scattering From Large Structures	1996
394850	Multiple-Model Approach To Finite Memory Adaptive Filtering	1992
749175	Statistical Computer-Aided Design For Microwave Circuits	1996
943999	Segmental Anhidrosis In The Spinal Dermatomes In Sjogrens Syndrome-Associated Neuropathy	1993
1121787	Rheological And Dielectrical Characterization Of Melt Mixed Polycarbonate-Multiwalled Carbon Nanotube Composites	2004
1177851	Explaining The Rate Spread On Corporate Bonds	2001
1256866	The Cyanobacterial Cell Division Factor Ftn6 Contains An N-Terminal Dnad-Like Domain	2009

False Negatives
Rank	Title	Year
259	Heterogeneity In Oscillator Networks: Are Smaller Worlds Easier To Synchronize?	2003
324	Assessing The Relevance Of Node Features For Network Structure	2009
385	The Use Of Edge-Betweenness Clustering To Investigate Biological Function In Protein Interaction Networks	2005
6605	A Measure Of Betweenness Centrality Based On Random Walks	2005
19863	On Decomposition Of Networks In Minimally Interconnected Subnetworks	1969
59900	Objective Criteria For Evaluation Of Clustering Methods	1971
139178	Optimization With Extremal Dynamics	2001
250583	The Tie Effect On Information Dissemination: The Spread Of A Commercial Rumor In Hong Kong	2002
281952	Compartments Revealed In Food-Web Structure	2003
1203248	Dynamic Asset Trees And Portfolio Analysis	2002