Skip to main content
SpringerLink
Log in

A novel self-attention deep subspace clustering

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

Most of the existing deep subspace clustering methods leverage convolutional autoencoders to obtain feature representation for non-linear data points. These methods commonly adopt the structure of a few convolutional layers because stacking many convolutional layers may cause computationally inefficient and optimization difficulties. However, long-range dependencies can hardly be captured when convolutional operations are not repeated enough, thus affect the quality of feature extraction which the performance of deep subspace clustering method highly lies in. To deal with this issue, we propose a novel self-attention deep subspace clustering (SADSC) model, which learns more favorable data representations by introducing self-attention mechanisms into convolutional autoencoders. Specifically, SADSC leverages three convolutional layers and add the self-attention layers after the first and third ones in encoders, then decoders have symmetric structures. The self-attention layers maintain the variable input sizes and can be easily combined with different convolutional layers in autoencoder. Experimental results on the handwritten recognition, face and object clustering datasets demonstrate the advantages of SADSC over the state-of-the-art deep subspace clustering models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Cao X, Zhang C, Fu H, Liu S, Zhang H (2015) Diversity-induced multi-view subspace clustering. CVPR

  2. Zhu W, Lu J, Zhou J (2018) Nonlinear subspace clustering for image clustering. Pattern Recogn Lett 107: 131–136.

  3. F. Lauer and C. Schn¨orr (2009) Spectral clustering of linear subspaces for motion segmentation. ICCV.

  4. Rao S, Tron R, Vidal R, Ma Y (2010) Motion segmentation in the presence of outlying, incomplete, or corrupted trajectories. IEEE Trans Pattern Anal Mach Intell 32(10):1832–1845

    Article  Google Scholar 

  5. Javed S, Mahmood A, Bouwmans T (2017) Background-foreground modeling based on spatiotemporal sparse subspace clustering. IEEE Trans Image Process 26(12):5840–5854

    Article  MathSciNet  Google Scholar 

  6. Wold S, Esbensen K, Geladi P (1987) Principal component analysis. Chemometrics Intell Lab Syst 2(1-3):37–52

  7. Hofmann T, Schölkopf B, Smola AJ (2008) Kernel methods in machine learning. Ann Stat 36(3):1171–1220

    Article  MathSciNet  Google Scholar 

  8. Ng AY, Jordan MI, Weiss Y (2002) On spectral clustering: Analysis and an algorithm. In: Proc. Adv. Neural Inf. Process. Syst. pp. 849-856.

  9. Ding S, Cong L, Hu Q, Jia H, Shi Z (2019) A multiway p-spectral clustering algorithm. Knowl Based Syst 164:371–377

    Article  Google Scholar 

  10. Wang L, Ding S, Jia H (2019) An improvement of spectral clustering via message passing and density sensitive similarity. IEEE Access 7:101054–101062

    Article  Google Scholar 

  11. Jia H, Ding S, Du M (2017) A nyström spectral clustering algorithm based on probability incremental sampling. Soft Comput 21(19):5815–5827

    Article  Google Scholar 

  12. Elhamifar E, Vidal R (2013) Sparse subspace clustering: algorithm, theory, and applications. TPAMI 35(11):2765–2781

    Article  Google Scholar 

  13. Peng X, Zhang L, Yi Z (2013) Scalable sparse subspace clustering. CVPR

  14. Soltanolkotabi M, Candes EJ et al (2012) A geometric analysis of subspace clustering with outliers. Ann Stat 40(4):2195–2238

    Article  MathSciNet  Google Scholar 

  15. You C, Robinson D, Vidal R (2016) Scalable sparse subspace clustering by orthogonal matching pursuit. CVPR

  16. You C, Vidal R (2015) Geometric conditions for subspace sparse recovery. ICML.

  17. Patel VM, Vidal R (2014) Kernel sparse subspace clustering. IEEE Int Conf Image Process. 2849-2853.

  18. Yin M, Guo Y, Gao J, He Z, Xie S (2016) Kernel sparse subspace clustering on symmetric positive definite manifolds. In: IEEE Conference on Computer Vision and Pattern Recognition. 5157–5164

  19. Zeng K, Ding S (2019) Single image super resolution using a polymorphic parallel CNN. Appl Intell 49(1):292–300

    Article  Google Scholar 

  20. Ding S, Du P, Zhao X,. Zhu Q, Xue Y (2019) BEMD image fusion based on PCNN and compressed sensing. Soft Comput 23(20):10045–10054.

  21. Dilokthanakul N, Mediano PA, Garnelo M, Lee MC, Salimbeni H, Arulkumaran K, Shanahan M (2016) Deep unsupervised clustering with Gaussian mixture variational autoencoders. arXiv preprint arXiv:1611.02648

  22. Ghasedi Dizaji K, Herandi A, Deng C, Cai W, Huang H (2017) Deep clustering via joint convolutional autoencoder embedding and relative entropy minimization. ICCV

  23. Kazemi H, Soleymani S, Taherkhani F, Iranmanesh S, Nasrabadi N (2018) Unsupervised image-to-image translation using domain-specific variational information bound. NIPS

  24. Tian F, Gao B, Cui Q, Chen E, Liu T-Y (2014) Learning deep representations for graph clustering. AAAI

  25. Xie J, Girshick R, Farhadi A (2016) Unsupervised deep embedding for clustering analysis. ICML

  26. Ji P, Zhang T, Li H, Salzmann M, Reid I (2017) Deep subspace clustering networks. NeurIPS

  27. Peng X, Xiao S, Feng J, Yau W-Y, Yi Z (2016) Deep subspace clustering with sparsity prior. IJCAI.

  28. Yang X, Deng C, Zheng F, Yan J, Liu W (2019) Deep spectral clustering using dual autoencoder network. arXiv preprint arXiv:1904.13113

  29. Masci J, Meier U, Cires¸an D, Schmidhuber J (2011) Stacked convolutional auto-encoders for hierarchical feature extraction. ICANN

  30. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B,Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. NeurIPS

  31. Cheng B, Liu G, Wang J, Huang Z, Yan S (2011) Multi-task low-rank affinity pursuit for image segmentation. IEEE ICCV, pages 2439–2446. IEEE

  32. Liu G, Lin Z, Yu Y (2010) Robust subspace segmentation by low-rank representation. ICML

  33. Vidal R, Favaro P (2014) Low rank subspace clustering (LRSC). Pattern Recogn Lett 43:47–61

    Article  Google Scholar 

  34. Ji P, Salzmann M, Li H (2014) Efficient dense subspace clustering (EDSC). In: IEEE Winter Conference on Applications of Computer Vision, pages 461–468. IEEE

  35. Lu C, Min H, Zhao Z, Zhu L, Huang D, Yan S (2012) Robust and efficient subspace segmentation via least squares regression. ECCV, pages 347–360

  36. Zhou P, Hou Y, Feng J (2018) Deep adversarial subspace clustering. CVPR

  37. Zhang J, Li C-G, You C, Qi X, Zhang H, Guo J, Lin Z (2019) Self-supervised convolutional subspace clustering network. CVPR

  38. Erxue M, Guo X, Liu Q (2018) A survey of clustering with deep learning: from the perspective of network architecture. IEEE Access 6:39501–39514

    Article  Google Scholar 

  39. Hinton GE, Salakhutdinov RR (2006) Reducing the dimensionality of data with neural networks. Science 313(5786):504–507

    Article  MathSciNet  Google Scholar 

  40. Vincent P, Larochelle H, Lajoie I, Bengio Y, Manzagol P-A (2010) Stacked denoising autoencoders:learning useful representations in a deep network with a local denoising criterion. JMLR 11(Dec):3371–3408

  41. Li F, Qiao H, Zhang B (2018) Discriminatively boosted image clustering with fully convolutional auto-encoders. Pattern Recogn

  42. Yang B, Fu X, Sidiropoulos ND, Hong M (2016) Towards k-means-friendly spaces: simultaneous deep learning and clustering

  43. Huang P, Huang Y, Wang W, Wang L (2014) Deep embedding network for clustering. In: Proc. 22nd Int. Conf. Pattern Recognit. (ICPR), pp. 1532–1537.

  44. Chen D, Lv J, Yi Z (2017) Unsupervised multi-manifold clustering by learning deep representation. In: Proc. Workshops 31st AAAI Conf. Artif. Intell. (AAAI), pp. 385–391.

  45. Shahand SA, Koltun V (2018) Deep continuous clustering. https://arxiv.org/abs/1803.01449

  46. Zhang H, Goodfellow I, Metaxas D (2019) Self-attention generative adversarial networks. In: International conference on machine learning (PMLR), pp. 7354–7363

  47. LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Article  Google Scholar 

  48. Vinh N, Epps J, Bailey J (2010) Information theoretic measures for clusterings comparison: variants, properties, normalization and correction for chance. JMLR 11:2837–2854

    MathSciNet  MATH  Google Scholar 

  49. Manning C, Raghavan P, Schutze H (2010) Introduction to information retrieval. Cambridge University Press

  50. Samaria F, Harter A (1994) Parameterisation of a stochastic model for human face identification. In: IEEE Workshop on Applications of Computer Vision, pages 138–142. IEEE

  51. Georghiades AS, Belhumeur PN, Kriegman DJ (2001) From few to many: Illumination cone models for face recognition under variable lighting and pose. IEEE TPAMI 23(6):643–660

    Article  Google Scholar 

  52. Nene S A, Nayar S K, Murase H (1996) Columbia object image library (COIL-20 and COIL-100)

Download references

Acknowledgements

This work is supported by the National Natural Science Foundations of China (No. 61976216 and No. 61672522).

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, China University of Mining and Technology, Xuzhou, 221116, China

    Zhengfan Chen, Shifei Ding & Haiwei Hou

  2. Mine Digitization Engineering Research Center of Ministry of Education of the People’s Republic of China, Xuzhou, 221116, China

    Shifei Ding

Authors
  1. Zhengfan Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shifei Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Haiwei Hou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shifei Ding.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, Z., Ding, S. & Hou, H. A novel self-attention deep subspace clustering. Int. J. Mach. Learn. & Cyber. 12, 2377–2387 (2021). https://doi.org/10.1007/s13042-021-01318-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-021-01318-4

Keywords

Search

Navigation