Skip to main content
SpringerLink
Log in

Dense capsule networks with fewer parameters

  • Foundations
  • Published:
Soft Computing Aims and scope Submit manuscript

Abstract

The capsule network (CapsNet) is a promising model in computer vision. It has achieved excellent results on MNIST, but it is still slightly insufficient in real images. Deepening capsule architectures is an effective way to improve performance, but the computational cost hinders their development. To overcome parameter growth and build an efficient architecture, this paper proposes a tensor capsule layer based on multistage separable convolutions and a dense capsule architecture. Multistage separable convolutions can effectively reduce the parameters at the cost of a small performance loss. In the dense capsule architecture, the use of dense connections allows the capsule network to be deeper and easier to train. Combining these two can achieve a novel lightweight dense capsule network. Experiments show that this network uses only 0.05% of the parameters of the CapsNet, but the performance is improved by 8.25% on CIFAR10. In addition, the full tensor capsule method is proposed to solve the problem of capsule network parameters changing with image scale. Experiments prove that this method can keep the parameters unchanged while affecting the performance in a small amount. In order to lighten the fully connected capsule layer, a dynamic routing based on separable matrices is proposed. In addition to applying it to our models, this algorithm also compresses the CapsNet by 41.25% while losing only 0.47% performance on CIFAR10. The parameter utilization index is proposed to quantify the relationship between parameters and performance. To our knowledge, this is the first paper to study lightweight capsule network.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  • Cheng X, He J, He J, Xu H (2019) Cv-CapsNet: complex-valued capsule network. IEEE Access 7:85492–85499. https://doi.org/10.1109/ACCESS.2019.2924548

    Article  Google Scholar 

  • Choi J, Seo H, Im S, Kang M (2019) Attention routing between capsules. In: IEEE/CVF international conference on computer vision workshop (ICCVW), pp 1981–1989. https://doi.org/10.1109/ICCVW.2019.00247

  • Chollet F (2017) Xception: Deep learning with depthwise separable convolutions. In: IEEE conference on computer vision and pattern recognition, pp 1800–1807. https://doi.org/10.1109/CVPR.2017.195

  • Deliege A, Cioppa A, Droogenbroeck MV (2018) Hitnet: a neural network with capsules embedded in a hit-or-miss layer, extended with hybrid data augmentation and ghost capsules. ArXiv preprint arXiv:1806.06519

  • He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: IEEE conference on computer vision and pattern recognition, pp 770–778. https://doi.org/10.1109/CVPR.2016.90

  • Hinton GE, Sabour S, Frosst N (2018) Matrix capsules with em routing. In: International conference on learning representations. https://openreview.net/forum?id=HJWLfGWRb

  • Hoogi A, Wilcox BM, Gupta Y, Rubin DL (2019) Self-attention capsule networks for image classification. ArXiv preprint arXiv:1904.12483

  • Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) Mobilenets: efficient convolutional neural networks for mobile vision applications. ArXiv preprint arXiv:1704.04861

  • Huang G, Liu Z, van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: IEEE conference on computer vision and pattern recognition, pp 2261–2269. https://doi.org/10.1109/CVPR.2017.243

  • Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. ArXiv preprint arXiv:1502.03167

  • Kingma DP, Ba J (2014) ADAM: a method for stochastic optimization. ArXiv preprint arXiv:1412.6980

  • Kosiorek A, Sabour S, Teh YW, Hinton GE (2019) Stacked capsule autoencoders. Adv Neural Inf Process Syst 32:15512–15522

    Google Scholar 

  • Krizhevsky A (2009) Learning multiple layers of features from tiny images. Tech. rep., University Toronto, Toronto

  • Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 25:1097–1105

    Google Scholar 

  • LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324. https://doi.org/10.1109/5.726791

    Article  Google Scholar 

  • Lin M, Chen Q, Yan S (2013) Network in network. ArXiv preprint arXiv:1312.4400

  • Marchisio A, Bussolino B, Colucci A, Hanif MA, Martina M, Masera G, Shafique M (2019) X-traincaps: accelerated training of capsule nets through lightweight software optimizations. ArXiv preprint arXiv:1905.10142

  • Mukhometzianov R, Carrillo J (2018) Capsnet comparative performance evaluation for image classification. ArXiv preprint arXiv:1606.07356

  • Paik I, Kwak T, Kim I (2019) Capsule networks need an improved routing algorithm. ArXiv preprint arXiv:1907.13327

  • Peer D, Stabinger S, Rodriguez-Sanchez A (2019) Limitations of routing-by-agreement based capsule networks. ArXiv preprint arXiv:1905.08744

  • Phaye SSR, Sikka A, Dhall A, Bathula DR (2018) Dense and diverse capsule networks: making the capsules learn better. ArXiv preprint arXiv:1805.04001

  • Rajasegaran J, Jayasundara V, Jayasekara S, Jayasekara H, Seneviratne S, Rodrigo R (2019) Deepcaps: going deeper with capsule networks. In: IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 10725–10733. https://doi.org/10.1109/CVPR.2019.01098

  • Ren H, Su J, Lu H (2019a) Evaluating generalization ability of convolutional neural networks and capsule networks for image classification via top-2 classification. ArXiv preprint arXiv:1901.10112

  • Ren Q, Shang S, He L (2019b) Adaptive routing between capsules. ArXiv preprint arXiv:1911.08119

  • Rosario VMd, Borin E, Breternitz M (2019) The multi-lane capsule network. IEEE Signal Process Lett 26(7):1006–1010. https://doi.org/10.1109/LSP.2019.2915661

    Article  Google Scholar 

  • Sabour S, Frosst N, Hinton GE (2017) Dynamic routing between capsules. Adv Neural Inf Process Syst 30:3856–3866

    Google Scholar 

  • Srivastava RK, Greff K, Schmidhuber J (2015) Training very deep networks. Adv Neural Inf Process Syst 28:2377–2385

    Google Scholar 

  • Sun K, Yuan L, Xu H, Wen X (2020) Deep tensor capsule network. IEEE. Access 8:96920–96933. https://doi.org/10.1109/ACCESS.2020.2996282

    Article  Google Scholar 

  • Wang D, Liu Q (2018) An optimization view on dynamic routing between capsules. In: International conference on learning representations workshop. https://openreview.net/forum?id=HJjtFYJDf

  • Xi E, Bing S, Jin Y (2017) Capsule network performance on complex data. ArXiv preprint arXiv:1712.03480

  • Xiang C, Zhang L, Tang Y, Zou W, Xu C (2018) Ms-CapsNet: a novel multi-scale capsule network. IEEE Signal Process Lett 25(12):1850–1854. https://doi.org/10.1109/LSP.2018.2873892

    Article  Google Scholar 

  • Xiao H, Rasul K, Vollgraf R (2017) Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. ArXiv preprint arXiv:1708.07747

  • Yin J, Li S, Zhu H, Luo X (2019) Hyperspectral image classification using CapsNet with well-initialized shallow layers. IEEE Geosci Remote Sens Lett 16(7):1095–1099. https://doi.org/10.1109/LGRS.2019.2891076

    Article  Google Scholar 

  • Zhang S, Zhao W, Wu X, Zhou Q (2019) Fast dynamic routing based on weighted kernel density estimation. Pract Exp, Concurr Comput. https://doi.org/10.1002/cpe.5281

  • Zhao Z, Kleinhans A, Sandhu G, Patel I, Unnikrishnan KP (2019) Capsule networks with max–min normalization. ArXiv preprint arXiv:1903.09662

Download references

Acknowledgements

This study was supported by the National Natural Science Foundation of China under Grant 61472278 and Major project of Tianjin under Grant 18ZXZNGX00150, and the Key Project of Natural Science Foundation of Tianjin University under Grant 2017ZD13, and the Research Project of Tianjin Municipal Education Commission under Grant 2017KJ255, and Natural Science Foundation of Tianjin under Grant 18JCYBJC84800.

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, Tianjin University of Technology, Tianjin, 300384, China

    Kun Sun, Xianbin Wen, Liming Yuan & Haixia Xu

  2. Key Laboratory of Computer Vision and System, Ministry of Education, Tianjin, 300384, China

    Kun Sun, Xianbin Wen, Liming Yuan & Haixia Xu

  3. Tianjin Key Laboratory of Intelligence Computing and Novel Software Technology, Tianjin, 300384, China

    Kun Sun, Xianbin Wen, Liming Yuan & Haixia Xu

Authors
  1. Kun Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xianbin Wen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Liming Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Haixia Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xianbin Wen.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sun, K., Wen, X., Yuan, L. et al. Dense capsule networks with fewer parameters. Soft Comput 25, 6927–6945 (2021). https://doi.org/10.1007/s00500-021-05774-6

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00500-021-05774-6

Keywords

Search

Navigation