Skip to main content
SpringerLink
Log in

Image saliency detection via multi-scale iterative CNN

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Salient object detection has received increasingly more attention and achieved significant progress lately due to the powerful features learned by deep convolutional neural networks (CNNs). In this work, we propose a multi-scale iterative CNN for salient object detection, which has two complementary subnetworks at different spatial scales. For each subnetwork, we augment the CNN structures with an iterative learning process to learn the saliency map, where early stages of the CNN give a rough estimate of the saliency map and the remaining errors are gradually learned to refine the saliency map. By merging predictions of the two subnetworks, the training error can be reduced significantly and the estimated saliency map becomes more accurate. Unlike some previous CNN-based methods which often rely on superpixel segmentations, the proposed model is fully CNN and hence can estimate the saliency map much more efficiently. Extensive experiments on standard benchmarks demonstrate that our method outperforms some of the state-of-the-art methods in terms of both accuracy and speed and achieves as good performance as some recent state-of-the-art end-to-end methods under fair settings.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. Note that iterative error correction has been a common practice in other fields such as automatic control and signal processing for decades.

  2. Essentially S is a weighted sum of output feature maps of all convolutional filters at each stage.

  3. Note that result of MC method [19] on MSRA-10k is not reported, since it is fine-tuned on MSRA-10k (after pretraining) with an 8000/2000 train/validation split.

  4. Note that DISC is also trained on MSRA-10k with a 9000/1000 split, so we do not compare with it on MSRA-10k dataset but on ECSSD, SED1 and Pascal-1500.

  5. The result of non-CRF version of DSS is not provided.

References

  1. Wu, R., Yu, Y., Wang, W.: Scale: supervised and cascaded Laplacian eigenmaps for visual object recognition based on nearest neighbors. In: CVPR, pp. 867–874 (2013)

  2. Lv, X., Zou, D., Zhang, L., Jia, S.: Feature coding for image classification based on saliency detection and fuzzy reasoning and its application in elevator videos. WSEAS Trans. Comput. 13(1), 266–276 (2014)

    Google Scholar 

  3. Wei, Y., Liang, X., Chen, Y., Shen, X., Cheng, M.-M., Feng, J., Zhao, Y., Yan, S.: Stc: a simple to complex framework for weakly-supervised semantic segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39(11), 2314–2320 (2017)

    Article  Google Scholar 

  4. Itti, L., Koch, C., Niebur, E.: A model of saliency-based visual attention for rapid scene analysis. IEEE Trans. Pattern Anal. Mach. Intell. 20(11), 1254–1259 (1998)

    Article  Google Scholar 

  5. Bharath, R., Jian Nicholas, L.Z., Cheng, X.: Scalable scene understanding using saliency-guided object localization. In: 2013 10th IEEE International Conference on Control and Automation (ICCA), pp. 1503–1508 (2013)

  6. Hadizadeh, H., Bajic, I.V.: Saliency-aware video compression. IEEE Trans. Image Process. 23(1), 19–33 (2014)

    Article  MathSciNet  Google Scholar 

  7. Liu, T., Yuan, Z., Sun, J., Wang, J., Zheng, N., Tang, X., Shum, H.-Y.: Learning to detect a salient object. IEEE Tran. Pattern Anal. Mach. Intell. 33(2), 353–367 (2011)

    Article  Google Scholar 

  8. Shen, X., Wu, Y.: A unified approach to salient object detection via low rank matrix recovery. In: CVPR, IEEE, pp. 853–860 (2012)

  9. Judd, T., Ehinger, K., Durand, F., Torralba, A.: Learning to predict where humans look. In: ICCV, IEEE, pp. 2106–2113 (2009)

  10. Perazzi, F., Krähenbühl, P., Pritch, Y., Hornung, A.: Saliency filters: contrast based filtering for salient region detection. In: CVPR, IEEE, pp. 733–740 (2012)

  11. Cheng, M., Mitra, N.J., Huang, X., Torr, P.H.S., Hu, S.: Global contrast based salient region detection. IEEE Trans. Pattern Anal. Mach. Intell. 37(3), 569–582 (2015)

    Article  Google Scholar 

  12. Zhu, W., Liang, S., Wei, Y., Sun, J.: Saliency optimization from robust background detection. In: CVPR, IEEE, pp. 2814–2821 (2014)

  13. Jiang, H., Wang, J., Yuan, Z., Wu, Y., Zheng, N., Li, S.: Salient object detection: a discriminative regional feature integration approach. In: CVPR, IEEE, pp. 2083–2090 (2013)

  14. Borji, A., Cheng, M.-M., Jiang, H., Li, J.: Salient object detection: a benchmark. IEEE Trans. Image Process. 24(12), 5706–5722 (2015)

    Article  MathSciNet  Google Scholar 

  15. Razavian, A.S., Azizpour, H., Sullivan, J., Carlsson, S.: Cnn features off-the-shelf: an astounding baseline for recognition. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 512–519 (2014)

  16. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778 (2016)

  17. Li, G., Yu, Y.: Visual saliency based on multiscale deep features. In: CVPR, IEEE, pp. 5455–5463 (2015)

  18. Wang, L., Lu, H., Ruan, X., Yang, M.-H.: Deep networks for saliency detection via local estimation and global search. In: CVPR, IEEE, pp. 3183–3192 (2015)

  19. Zhao, R., Ouyang, W., Li, H., Wang, X.: Saliency detection by multi-context deep learning. In: CVPR, IEEE, pp. 1265–1274 (2015)

  20. Chen, T., Lin, L., Liu, L., Luo, X., Li, X.: Disc: deep image saliency computing via progressive representation learning. IEEE Trans. Neural Netw. Learn. Syst. PP(99), 1–15 (2016)

    MathSciNet  Google Scholar 

  21. Yan, Q., Xu, L., Shi, J., Jia, J.: Hierarchical saliency detection. In: CVPR, IEEE, pp. 1155–1162 (2013)

  22. Zeiler, M. D., Fergus, R.: Visualizing and understanding convolutional networks. In: ECCV, Springer, pp. 818–833 (2014)

  23. Srivastava, R. K, Greff, K., Schmidhuber, J.: Training very deep networks. In: Advances in Neural Information Processing Systems, pp. 2368–2376 (2015)

  24. Zhang, J., Shan, S., Kan, M., Chen, X.: Coarse-to-fine auto-encoder networks (CFAN) for real-time face alignment. In: ECCV, Springer, pp. 1–16 (2014)

  25. Carreira, J., Agrawal, P., Fragkiadaki, K., Malik, J.: Human pose estimation with iterative error feedback. In: CVPR, pp. 4733–4742 (2016)

  26. Achanta, R., Hemami, S., Estrada, F., Susstrunk, S.: Frequency-tuned salient region detection. In: CVPR, IEEE, pp. 1597–1604 (2009)

  27. Hou, X., Zhang, L.: Saliency detection: a spectral residual approach. In: CVPR, pp. 1–8 (2007)

  28. Goferman, S., Zelnik-Manor, L., Tal, A.: Context-aware saliency detection. IEEE Trans. Pattern Anal. Mach. Intell. 34(10), 1915–1926 (2012)

    Article  Google Scholar 

  29. Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T.: Caffe: convolutional architecture for fast feature embedding. In: Proceedings of the ACM International Conference on Multimedia, ACM, pp. 675–678 (2014)

  30. Farabet, C., Couprie, C., Najman, L., LeCun, Y.: Learning hierarchical features for scene labeling. IEEE Trans. Pattern Anal. Mach. Intell. 35(8), 1915–1929 (2013)

    Article  Google Scholar 

  31. Huang, X., Shen, C., Boix, X., Zhao, Q.: Salicon: reducing the semantic gap in saliency prediction by adapting deep neural networks. In: ICCV, pp. 262–270 (2015)

  32. Chen, L.-C., Yang, Y., Wang, J., Xu, W., Yuille, A. L: Attention to scale: Scale-aware semantic image segmentation. In: CVPR, pp. 3640–3649 (2016)

  33. Liu, N., Han, J., Yang, M.-H.: Picanet: learning pixel-wise contextual attention for saliency detection. In: CVPR (2018)

  34. Wang, T., Zhang, L., Wang, S., Lu, H., Yang, G., Ruan, X., Borji, A.: Detect globally, refine locally: a novel approach to saliency detection. In: CVPR (2018)

  35. Wang, T., Borji, A., Zhang, L., Zhang, P., Lu, H.: A stagewise refinement model for detecting salient objects in images. In: ICCV (2017)

  36. Zhang, P., Wang, D., Lu, H., Wang, H., Yin, B.: Learning uncertain convolutional features for accurate saliency detection. In: ICCV (2017)

  37. Chen, X., Zheng, A., Li, J., Lu, F.: Look, perceive and segment: finding the salient objects in images via two-stream fixation-semantic CNNS. In: ICCV (2017)

  38. Luo, Z., Mishra, A., Achkar, A., Eichel, J., Li, S., Jodoin, P.-M.: Non-local deep features for salient object detection. In: CVPR (2017)

  39. Hariharan, B., Arbeláez, P., Girshick, R., Malik, J.: Hypercolumns for object segmentation and fine-grained localization. In: CVPR, IEEE, pp. 447–456 (2015)

  40. Li, Guanbin, Y., Y.: Deep contrast learning for salient object detection. In: CVPR, pp. 478–487 (2016)

  41. Hu, P., Shuai, B., Liu, J., Wang, G.: Deep level sets for salient object detection. In: CVPR (2017)

  42. Zhang, P., Wang, D., Lu, H., Wang, H., Ruan, X.: Amulet: aggregating multi-level convolutional features for salient object detection. In: ICCV, pp. 202–211 (2017)

  43. Hou, Q., Cheng, M.-M., Hu, X., Borji, A., Tu, Z., Torr, P. H.: Deeply supervised salient object detection with short connections. In: CVPR, pp. 3203–3212 (2017)

  44. Zhang, L., Dai, J., Lu, H., He, Y., Wang, G.: A bi-directional message passing model for salient object detection. In: CVPR, pp. 1741–1750 (2018)

  45. Deng, Z., Hu, X., Zhu, L., Xu, X., Qin, J., Han, G., Heng, P.-A.: R3net: recurrent residual refinement network for saliency detection. In: IJCAI, AAAI Press, pp. 684–690 (2018)

  46. Hu, X., Zhu, L., Qin, J., Fu, C.-W., Heng, P.-A.: Recurrently aggregating deep features for salient object detection. In: AAAI (2018)

  47. Li, X., Yang, F., Cheng, H., Liu, W., Shen, D.: Contour knowledge transfer for salient object detection. In: ECCV, pp. 355–370 (2018)

  48. Chen, S., Tan, X., Wang, B., Hu, X.: Reverse attention for salient object detection. In: ECCV, pp. 234–250 (2018)

  49. Li, G., Xie, Y., Lin, L., Yu, Y.: Instance-level salient object segmentation. In: CVPR (2017)

  50. Hou, Q., Cheng, M.-M., Hu, X., Borji, A., Tu, Z., Torr, P. H.: Deeply supervised salient object detection with short connections. IEEE Trans. Pattern Anal. Mach. Intell. 41(1), 1–1

  51. Xie, S., Tu, Z.: Holistically-nested edge detection. In: ICCV, pp. 1395–1403 (2015)

  52. Zhao, T., Wu, X.: Pyramid feature attention network for saliency detection. In: CVPR (2019)

  53. Li, Z., Lang, C., Chen, Y., Liew, J., Feng, J.: Deep reasoning with multi-scale context for salient object detection. arXiv preprint arXiv:1901.08362 (2019)

  54. Zhou, S., Wang, J., Wang, F., Huang, D.: Se2net: siamese edge-enhancement network for salient object detection. arXiv preprint arXiv:1904.00048 (2019)

  55. Liu, Y., Fan, D.-P., Nie, G.-Y., Zhang, X., Petrosyan, V., Cheng, M.-M.: Dna: deeply-supervised nonlinear aggregation for salient object detection. arXiv preprint arXiv:1903.12476 (2019)

  56. Hu, X., Fu, C.-W., Zhu, L., Heng, P.-A.: Sac-net: spatial attenuation context for salient object detection. arXiv preprint arXiv:1903.10152 (2019)

  57. Alpert, S., Galun, M., Brandt, A., Basri, R.: Image segmentation by probabilistic bottom-up aggregation and cue integration. IEEE Trans. Pattern Anal. Mach. Intell. 34(2), 315–327 (2012)

    Article  Google Scholar 

  58. Zou, W., Kpalma, K., Liu, Z., Ronsin, J.: Segmentation driven low-rank matrix recovery for saliency detection. In: BMVC, pp. 1–13 (2013)

  59. Yang, C., Zhang, L., Lu, H., Ruan, X., Yang, M.-H.: Saliency detection via graph-based manifold ranking. In: CVPR, IEEE, pp. 3166–3173 (2013)

  60. Wei, Y., Wen, F., Zhu, W., Sun, J.: Geodesic saliency using background priors. In: ECCV, Springer, pp. 29–42 (2012)

  61. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

Download references

Author information

Authors and Affiliations

  1. Shanghai Institute of Microsysterm and Information Technology, Chinese Academy of Sciences, Shanghai, 200050, People’s Republic of China

    Kun Huang

  2. School of Information Science and Technology, ShanghaiTech University, Shanghai, 201210, People’s Republic of China

    Kun Huang & Shenghua Gao

  3. University of Chinese Academy of Sciences, Beijing, People’s Republic of China

    Kun Huang

Authors
  1. Kun Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shenghua Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kun Huang.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huang, K., Gao, S. Image saliency detection via multi-scale iterative CNN. Vis Comput 36, 1355–1367 (2020). https://doi.org/10.1007/s00371-019-01734-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-019-01734-2

Keywords

Search

Navigation