Deep neural network models have achieved state-of-the-art results in various tasks related to vision and/or language. Despite the use of large training data, most models are trained by iterating over single input-output pairs, discarding the remaining examples for the current prediction. In this work, we actively exploit the training data to improve the robustness and interpretability of deep neural networks, using the information from nearest training examples to aid the prediction both during training and testing. Specifically, the proposed approach uses the target of the nearest input example to initialize the memory state of an LSTM model or to guide attention mechanisms. We apply this approach to image captioning and sentiment analysis, conducting experiments with both image and text retrieval. Results show the effectiveness of the proposed models for the two tasks, on the widely used Flickr8 and IMDB datasets, respectively. Our code is publicly available http://github.com/RitaRamo/retrieval-augmentation-nn.

中文翻译：

---

检索增强以提高深度神经网络的鲁棒性和可解释性

深度神经网络模型已在与视觉和/或语言相关的各种任务中取得了最先进的结果。尽管使用了大量的训练数据，但是大多数模型都是通过迭代单个输入输出对来训练的，并丢弃了当前预测的其余示例。在这项工作中，我们积极利用训练数据来改善深度神经网络的鲁棒性和可解释性，并使用最近的训练示例中的信息来辅助训练和测试过程中的预测。具体而言，所提出的方法使用最近的输入示例的目标来初始化LSTM模型的内存状态或指导注意机制。我们将这种方法应用于图像字幕和情感分析，并通过图像和文本检索进行实验。结果表明，分别在广泛使用的Flickr8和IMDB数据集上，针对两个任务提出的模型是有效的。我们的代码可从http://github.com/RitaRamo/retrieval-augmentation-nn公开获得。