Skip to main content
SpringerLink
Log in

Deep learning-based data augmentation method and signature verification system for offline handwritten signature

  • Theoretical Advances
  • Published:
Pattern Analysis and Applications Aims and scope Submit manuscript

Abstract

Offline handwritten signature verification is a challenging pattern recognition task. One of the most significant limitations of the handwritten signature verification problem is inadequate data for training phases. Due to this limitation, deep learning methods that have obtained the state-of-the-art results in many areas achieve quite unsuccessful results when applied to signature verification. In this study, a new use of Cycle-GAN is proposed as a data augmentation method to address the inadequate data problem on signature verification. We also propose a novel signature verification system based on Caps-Net. The proposed data augmentation method is tested on four different convolutional neural network (CNN) methods, VGG16, VGG19, ResNet50, and DenseNet121, which are widely used in the literature. The method has provided a significant contribution to all mentioned CNN methods’ success. The proposed data augmentation method has the best effect on the DenseNet121. We also tested our data augmentation method with the proposed signature verification system on two widely used databases: GPDS and MCYT. Compared to other studies, our verification system achieved the state-of-the-art results on MCYT database, while it reached the second-best verification result on GPDS.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Afshar P, Mohammadi A, Plataniotis KN (2018) Brain tumor type classification via capsule networks. In: 2018 25th IEEE international conference on image processing (ICIP). IEEE, pp 3129–3133

  2. Ahmed E, Jones M, Marks TK (2015) An improved deep learning architecture for person re-identification. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3908–3916

  3. Alonso-Fernandez F, Fairhurst MC, Fierrez J, Ortega-Garcia J (2007) Automatic measures for predicting performance in off-line signature. In: 2007 IEEE international conference on image processing, vol 1. IEEE, pp I–369

  4. Alvarez G, Sheffer B, Bryant M (2016) Offline signature verification with convolutional neural networks. Technical report, Stanford University, Stanford

  5. Anguita D, Ridella S, Rivieccio F (2005) K-fold generalization capability assessment for support vector classifiers. In: Proceedings. 2005 IEEE international joint conference on neural networks, 2005, vol 2. IEEE, pp 855–858

  6. Ayan E, Ünver HM (2018) Data augmentation importance for classification of skin lesions via deep learning. In: 2018 Electric electronics, computer science, biomedical engineerings’ meeting (EBBT). IEEE, pp 1–4

  7. Bae YJ, Fairhurst MC (1995) Parallelism in dynamic time warping for automatic signature verification. In: Proceedings of the third international conference on document analysis and recognition, 1995, vol 1. IEEE, pp 426–429

  8. Becerra A, de la Rosa JI, González E, Pedroza AD, Martínez JM, Escalante NI (2017) Speech recognition using deep neural networks trained with non-uniform frame-level cost functions. In: 2017 IEEE international autumn meeting on power, electronics and computing (ROPEC), pp 1–6. https://doi.org/10.1109/ROPEC.2017.8261588

  9. Canuto J, Dorizzi B, Montalvão J, Matos L (2016) On the infinite clipping of handwritten signatures. Pattern Recognit Lett 79:38–43

    Article  Google Scholar 

  10. Chollet F et al (2015) Keras: deep learning library for theano and tensorflow. https://keras io/k7:8

  11. Costa AC, Oliveira HCR, Catani JH, de Barros N, Melo CFE, Vieira MAC (2018) Data augmentation for detection of architectural distortion in digital mammography using deep learning approach

  12. de SanRoman PP, Benois-Pineau J, Domenger JP, Paclet F, Cataert D, de Rugy A (2017) Saliency driven object recognition in egocentric videos with deep CNN: toward application in assistance to Neuroprostheses. Comput Vis Image Underst 164:82–91. https://doi.org/10.1016/j.cviu.2017.03.001

    Article  Google Scholar 

  13. Eskander GS, Sabourin R, Granger E (2013) Hybrid writer-independent-writer-dependent offline signature verification system. IET Biom 2(4):169–181

    Article  Google Scholar 

  14. Fahmy MMM (2010) Online handwritten signature verification system based on DWT features extraction and neural network classification. Ain Shams Eng J 1(1):59–70

    Article  Google Scholar 

  15. Ferrer MA, Diaz-Cabrera M, Morales A (2013) Synthetic off-line signature image generation. In: 2013 International conference on biometrics (ICB). IEEE, pp 1–7

  16. Fierrez-Aguilar J, Alonso-Hermira N, Moreno-Marquez G, Ortega-Garcia J (2004) An off-line signature verification system based on fusion of local and global information. In: International workshop on biometric authentication. Springer, pp 295–306

  17. Frid-Adar M, Diamant I, Klang E, Amitai M, Goldberger J, Greenspan H (2018) GAN-based synthetic medical image augmentation for increased CNN performance in liver lesion classification. Neurocomputing 321:321–331. https://doi.org/10.1016/j.neucom.2018.09.013

    Article  Google Scholar 

  18. Frosst N, Sabour S, Hinton G (2018) DARCCC: detecting adversaries by reconstruction from class conditional capsules. arXiv preprint arXiv:1811.06969

  19. Ge W, Yu Y (2017) Borrowing treasures from the wealthy: deep transfer learning through selective joint fine-tuning. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1086–1095

  20. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Advances in neural information processing systems, pp 2672–2680

  21. Goodfellow IJ, Shlens J, Szegedy C (2014) Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572

  22. Hafemann LG, Sabourin R, Oliveira LS (2016) Analyzing features learned for offline signature verification using deep CNNs. In: 2016 23rd international conference on pattern recognition (ICPR), pp 2989–2994. https://doi.org/10.1109/ICPR.2016.7900092

  23. Hafemann LG, Sabourin R, Oliveira LS (2017a) Learning features for offline handwritten signature verification using deep convolutional neural networks. Pattern Recognit 70:163–176. https://doi.org/10.1016/j.patcog.2017.05.012

    Article  Google Scholar 

  24. Hafemann LG, Sabourin R, Oliveira LS (2017b) Offline handwritten signature verification; literature review. In: 2017 Seventh international conference on image processing theory, tools and applications (IPTA), pp 1–8. https://doi.org/10.1109/IPTA.2017.8310112

  25. HaNoi V (2020) Interactive educational content using marker based augmented reality. Inf Technol Intell Transp Syst 323:77

    Google Scholar 

  26. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  27. Isola P, Zhu JY, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. arXiv preprint

  28. Justino EJR, El Yacoubi A, Bortolozzi F, Sabourin R (2000) An off-line signature verification system using HMM and graphometric features. In: Proceedings of the 4th international workshop on document analysis systems, Citeseer, pp 211–222

  29. Justino EJR, Bortolozzi F, Sabourin R (2005) A comparison of SVM and HMM classifiers in the off-line signature verification. Pattern Recognit Lett 26(9):1377–1385

    Article  Google Scholar 

  30. Khalajzadeh H, Mansouri M, Teshnehlab M (2012) Persian signature verification using convolutional neural networks. Int J Eng Res Technol 1(2):7–12

    Google Scholar 

  31. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  32. Larsson G, Maire M, Shakhnarovich G (2016) Fractalnet: ultra-deep neural networks without residuals. arXiv preprint arXiv:1605.07648

  33. Liwicki M, Malik MI, v d Heuvel CE, Chen X, Berger C, Stoel R, Blumenstein M, Found B (2011) Signature verification competition for online and offline skilled forgeries (SigComp2011). In: 2011 International conference on document analysis and recognition, pp 1480–1484, https://doi.org/10.1109/ICDAR.2011.294

  34. Liwicki M, Malik MI, Alewijnse L, v d Heuvel E, Found B (2012) ICFHR 2012 competition on automatic forensic signature verification (4NsigComp 2012). In: 2012 International conference on frontiers in handwriting recognition, pp 823–828. https://doi.org/10.1109/ICFHR.2012.217

  35. Lv JJ, Shao XH, Huang JS, Zhou XD, Zhou X (2017) Data augmentation for face recognition. Neurocomputing 230:184–196

    Article  Google Scholar 

  36. Maergner P, Howe N, Riesen K, Ingold R, Fischer A (2018) Offline signature verification via structural methods: graph edit distance and inkball models. In: 2018 16th international conference on frontiers in handwriting recognition (ICFHR). IEEE, pp 163–168

  37. Maergner P, Howe NR, Riesen K, Ingold R, Fischer A (2019) Graph-based offline signature verification. arXiv preprint arXiv:1906.10401

  38. Maergner P, Pondenkandath V, Alberti M, Liwicki M, Riesen K, Ingold R, Fischer A (2019) Combining graph edit distance and triplet networks for offline signature verification. Pattern Recognit Lett 125:527–533

    Article  Google Scholar 

  39. Malik MI, Liwicki M, Alewijnse L, Ohyama W, Blumenstein M, Found B (2013) ICDAR 2013 competitions on signature verification and writer identification for on- and offline skilled forgeries (SigWiComp 2013). In: 2013 12th international conference on document analysis and recognition, pp 1477–1483. https://doi.org/10.1109/ICDAR.2013.220

  40. Mao X, Li Q, Xie H, Lau RYK, Wang Z, Smolley SP (2017) Least squares generative adversarial networks. In: 2017 IEEE international conference on computer vision (ICCV). IEEE, pp 2813–2821

  41. Martin Sagayam K, Ho CC, Henesey L, Bestak R (2018) 3D scenery learning on solar system by using marker based augmented reality. In: 4th international conference of the virtual and augmented reality in education, VARE 2018; Budapest, Dime University of Genoa, pp 139–143

  42. Masoudnia S, Mersa O, Araabi BN, Vahabie AH, Sadeghi MA, Ahmadabadi MN (2019) Multi-representational learning for offline signature verification using multi-loss snapshot ensemble of CNNs. Expert Syst Appl 133:317–330

    Article  Google Scholar 

  43. Mobiny A, Van Nguyen H (2018) Fast capsnet for lung cancer screening. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 741–749

  44. Mukhometzianov R, Carrillo J (2018) CapsNet comparative performance evaluation for image classification. arXiv preprint arXiv:1805.11195

  45. Ooi SY, Teoh ABJ, Pang YH, Hiew BY (2016) Image-based handwritten signature verification using hybrid methods of discrete Radon transform, principal component analysis and probabilistic neural network. Appl Soft Comput 40:274–282. https://doi.org/10.1016/j.asoc.2015.11.039

    Article  Google Scholar 

  46. Ortega-Garcia J, Fierrez-Aguilar J, Simon D, Gonzalez J, Faundez-Zanuy M, Espinosa V, Satue A, Hernaez I, Igarza JJ, Vivaracho C et al (2003) MCYT baseline corpus: a bimodal biometric database. IEE Proc Vis Image Signal Process 150(6):395–401

    Article  Google Scholar 

  47. Otsu N (1979) A threshold selection method from gray-level histograms. IEEE Trans Syst Man Cybern 9(1):62–66

    Article  Google Scholar 

  48. Porwik P, Doroz R, Orczyk T (2016) Signatures verification based on PNN classifier optimised by PSO algorithm. Pattern Recognit 60:998–1014

    Article  Google Scholar 

  49. Prabhakar G, Kailath B, Natarajan S, Kumar R (2017) Obstacle detection and classification using deep learning for tracking in high-speed autonomous driving. In: 2017 IEEE region 10 symposium (TENSYMP), pp 1–6. https://doi.org/10.1109/TENCONSpring.2017.8069972

  50. Ribeiro B, Gonçalves I, Santos S, Kovacec A (2011) Deep learning networks for off-line handwritten signature recognition. In: Iberoamerican congress on pattern recognition. Springer, pp 523–532

  51. Sabour S, Frosst N, Hinton GE (2017) Dynamic routing between capsules. In: Advances in neural information processing systems, pp 3856–3866

  52. Sagayam KM, Hemanth DJ (2019) A probabilistic model for state sequence analysis in hidden Markov model for hand gesture recognition. Comput Intell 35(1):59–81

    Article  MathSciNet  Google Scholar 

  53. Sagayam KM, Ponraj DN, Winston J, Jeba E, Clara A (2019) Authentication of biometric system using fingerprint recognition with euclidean distance and neural network classifier. Project: hand posture and gesture recognition techniques for virtual reality applications: a survey

  54. Sagayama KM, Viyasb TV, Hoa CC, Heneseyb LE (2017) Virtual robotic arm control with hand gesture recognition and deep learning strategies. Deep Learn Image Process Appl 31:50

    Google Scholar 

  55. Shariatmadari S, Emadi S, Akbari Y (2019) Patch-based offline signature verification using one-class hierarchical deep learning. Int J Doc Anal Recognit 22:375–385. https://doi.org/10.1007/s10032-019-00331-2

    Article  Google Scholar 

  56. Sharif M, Khan MA, Faisal M, Yasmin M, Fernandes SL (2018) A framework for offline signature verification system: best features selection approach. Pattern Recognit Lett. https://doi.org/10.1016/j.patrec.2018.01.021

    Article  Google Scholar 

  57. Soleimani A, Araabi BN, Fouladi K (2016) Deep multitask metric learning for offline signature verification. Pattern Recognit Lett 80:84–90. https://doi.org/10.1016/j.patrec.2016.05.023

    Article  Google Scholar 

  58. Tustison NJ, Avants BB, Lin Z, Feng X, Cullen N, Mata JF, Flors L, Gee JC, Altes TA, Mugler JP III, Qing K (2018) Convolutional neural networks with template-based data augmentation for functional lung image quantification. Acad Radiol. https://doi.org/10.1016/j.acra.2018.08.003

    Article  Google Scholar 

  59. Uçar A, Demir Y, Güzeliş C (2016) Moving towards in object recognition with deep learning for autonomous driving applications. In: 2016 International symposium on INnovations in Intelligent SysTems and Applications (INISTA), pp 1–5. https://doi.org/10.1109/INISTA.2016.7571862

  60. Vasconcelos CN, Vasconcelos BN (2017) Increasing deep learning melanoma classification by classical and expert knowledge based image transforms. CoRR, arXiv:1702.07025

  61. Xiao X, Jin L, Yang Y, Yang W, Sun J, Chang T (2017) Building fast and compact convolutional neural networks for offline handwritten Chinese character recognition. Pattern Recognit 72:72–81. https://doi.org/10.1016/j.patcog.2017.06.032

    Article  Google Scholar 

  62. Xie S, Yang T, Wang X, Lin Y (2015) Hyper-class augmented and regularized deep learning for fine-grained image classification. In: The IEEE conference on computer vision and pattern recognition (CVPR)

  63. Xu Y, Jia R, Mou L, Li G, Chen Y, Lu Y, Jin Z (2016) Improved relation classification by deep recurrent neural networks with data augmentation. arXiv preprint arXiv:1601.03651

  64. Yapıcı MM, Tekerek A, Topaloglu N (2018) Convolutional neural network based offline signature verification application. In: 2018 International congress on big data. Deep learning and fighting cyber terrorism (IBIGDELFT). IEEE, pp 30–34

  65. Yapıcı MM, Tekerek A, Topaloğlu N (2019) Performance comparison of convolutional neural network models on GPU. In: 2019 IEEE 13th international conference on application of information and communication technologies (AICT). IEEE, pp 1–4

  66. Zhang L, Zhang L, Du B (2016) Deep learning for remote sensing data: a technical tutorial on the state of the art. IEEE Geosci Remote Sens Mag 4(2):22–40

    Article  Google Scholar 

  67. Zhang Y, Ling C (2018) A strategy to apply machine learning to small datasets in materials science. NPJ Comput Mater 4(1):25

    Article  Google Scholar 

  68. Zhu JY, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. arXiv preprint

  69. Zhuang H, Wang C, Li C, Wang Q, Zhou X (2017) Natural language processing service based on stroke-level convolutional networks for Chinese text classification. In: 2017 IEEE international conference on web services (ICWS), pp 404–411. https://doi.org/10.1109/ICWS.2017.46

  70. Zois EN, Alewijnse L, Economou G (2016) Offline signature verification and quality characterization using poset-oriented grid features. Pattern Recognit 54:162–177

    Article  Google Scholar 

Download references

Acknowledgements

This work has been supported by the NVIDIA Corporation. All experimental studies were carried out on the TITAN XP graphics card donated by NVIDIA. We sincerely thank NVIDIA Corporation for their supports.

Author information

Authors and Affiliations

  1. Computer Technologies Department, Ankara University, Ankara, Turkey

    Muhammed Mutlu Yapıcı

  2. Information Technology Department, Gazi University, Ankara, Turkey

    Adem Tekerek

  3. Computer Engineering Faculty, Technology Faculty, Gazi University, Ankara, Turkey

    Nurettin Topaloğlu

Authors
  1. Muhammed Mutlu Yapıcı
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Adem Tekerek
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nurettin Topaloğlu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Muhammed Mutlu Yapıcı.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yapıcı, M.M., Tekerek, A. & Topaloğlu, N. Deep learning-based data augmentation method and signature verification system for offline handwritten signature. Pattern Anal Applic 24, 165–179 (2021). https://doi.org/10.1007/s10044-020-00912-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10044-020-00912-6

Keywords

Search

Navigation