Deep convolutional neural network (CNN) has achieved great success in image restoration. However, previous methods ignore the complementarity between low-level and high-level features, thereby leading to limited image reconstruction quality. In this paper, we propose a two-stream sparse network (TSSN) to explicitly learn shallow and deep features to enforce their respective contribution to image restoration. The shallow stream learns shallow features (e.g., texture edges), and the deep stream learns deep features (e.g., salient semantics). In each stream, sparse residual block (SRB) is proposed to efficiently aggregate hierarchical features by constructing sparse connections among layers in the local block. Spatial-wise and channel-wise attention are used to fuse the shallow and deep stream which recalibrates features by weight assignment in both spatial and channel dimensions. A novel loss function called Softmax-$L_1$ loss is proposed to increase penalties of pixels that have large $L_1$ loss (i.e., hard pixels). TSSN is evaluated with three representative IR applications, i.e., single image super-resolution, image denoising and JPEG deblocking. Extensive experiments demonstrate that TSSN outperforms most of state-of-the-art methods on benchmark datasets on both quantitative metric and visual quality.

深度卷积神经网络（CNN）在图像复原方面取得了巨大成功。然而，先前的方法忽略了低级和高级特征之间的互补性，从而导致有限的图像重建质量。在本文中，我们提出了一种两流稀疏网络（TSSN），以明确学习浅层和深层特征，以增强它们各自对图像恢复的贡献。浅流学习浅特征（例如纹理边缘），深流学习深特征（例如，显着语义）。在每个流中，提出了稀疏残差块（SRB）以通过在局部块中的各层之间构造稀疏连接来有效地聚合层次结构特征。空间注意和通道注意被用于融合浅流和深流，这通过在空间和通道维度上的权重分配来重新校准特征。一种称为Softmax-的新型损耗函数${\mathrm{大号}}_{\mathrm{1个}}$ 建议增加损耗以增加具有较大像素的像素的惩罚 ${\mathrm{大号}}_{\mathrm{1个}}$损耗（即硬像素）。TSSN通过三种代表性的IR应用程序进行评估，即单图像超分辨率，图像去噪和JPEG解块。大量的实验表明，TSSN在定量指标和视觉质量上都优于基准数据集上的大多数最新方法。