Skip to main content
Log in

Estimating pose from pressure data for smart beds with deep image-based pose estimators

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

In-bed pose estimation has shown value in fields such as hospital patient monitoring, sleep studies, and smart homes. In this paper, we explore different strategies for detecting body pose from highly ambiguous pressure data, with the aid of pre-existing pose estimators. We examine the performance of pre-trained pose estimators by using them either directly or by re-training them on two pressure datasets. We also explore other strategies utilizing a learnable pre-processing domain adaptation step, which transforms the vague pressure maps to a representation closer to the expected input space of common purpose pose estimation modules. Accordingly, we used a fully convolutional network with multiple scales to provide the pose-specific characteristics of the pressure maps to the pre-trained pose estimation module. Our complete analysis of different approaches shows that the combination of learnable pre-processing module along with re-training pre-existing image-based pose estimators on the pressure data is able to overcome issues such as highly vague pressure points to achieve very high pose estimation accuracy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Achilles F, Ichim AE, Coskun H, Tombari F, Noachtar S, Navab N (2016) Patient mocap: human pose estimation under blanket occlusion for hospital monitoring applications. In: International conference on medical image computing and computer-assisted intervention, pp 491–499

  2. Andriluka M, Pishchulin L, Gehler P, Schiele B (2014) 2d human pose estimation: New benchmark and state of the art analysis. In: IEEE conference on computer Vision and Pattern Recognition (CVPR), pp 3686–3693

  3. Cao Z, Simon T, Wei SE, Sheikh Y (2017) Realtime multi-person 2d pose estimation using part affinity fields. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 1302–1310

  4. Casas L, Navab N, Demirci S (2019) Patient 3d body pose estimation from pressure imaging. Int J Comput Assist Radiol Surg 14(3):517–524

    Article  Google Scholar 

  5. Chen K, Gabriel P, Alasfour A, Gong C, Doyle WK, Devinsky O, Friedman D, Dugan P, Melloni L, Thesen T et al (2018a) Patient-specific pose estimation in clinical environments. IEEE J Transl Eng Health Med 1

  6. Chen Y, Shen C, Wei XS, Liu L, Yang J (2017) Adversarial posenet: A structure-aware convolutional network for human pose estimation. In: Proceedings of the IEEE international conference on computer vision (ICCV), pp 1212–1221

  7. Chen Y, Wang Z, Peng Y, Zhang Z, Yu G, Sun J (2018b) Cascaded pyramid network for multi-person pose estimation. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 7103–7112

  8. Chu X, Yang W, Ouyang W, Ma C, Yuille AL, Wang X (2017) Multi-context attention for human pose estimation. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 5669–5678

  9. Clever HM, Kapusta A, Park D, Erickson Z, Chitalia Y, Kemp CC (2018) 3d human pose estimation on a configurable bed from a pressure image. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), pp 54–61

  10. Clever HM, Erickson Z, Kapusta A, Turk G, Liu K, Kemp CC (2020) Bodies at rest: 3d human pose and shape estimation from a pressure image using synthetic data. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 6215–6224

  11. Cunha JPS, Choupina HMP, Rocha AP, Fernandes JM, Achilles F, Loesch AM, Vollmar C, Hartl E, Noachtar S (2016) Neurokinect: a novel low-cost 3d video-eeg system for epileptic seizure motion quantification. PLoS One 11(1):e0145669

    Article  Google Scholar 

  12. Davoodnia V, Etemad A (2019) Identity and posture recognition in smart beds with deep multitask learning. In: IEEE international conference on systems, man and cybernetics (SMC), pp 3054–3059

  13. Davoodnia V, Slinowsky M, Etemad A (2020) Deep multitask learning for pervasive bmi estimation and identity recognition in smart beds. J Ambient Intell Humaniz Comput :1–15

  14. Davoodnia V, Ghorbani S, Etemad A (2021) In-bed pressure-based pose estimation using image space representation learning. In: IEEE international conference on acoustics, speech and signal processing (ICASSP)

  15. Goldberger AL, Amaral LA, Glass L, Hausdorff JM, Ivanov PC, Mark RG, Mietus JE, Moody GB, Peng CK, Stanley HE (2000) Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals. Circulation 101(23):e215–e220

    Article  Google Scholar 

  16. Goldberger AL, Amaral LAN, Glass L, Hausdorff JM, Ivanov PC, Mark RG, Mietus JE, Moody GB, Peng CK, Stanley HE (2000) PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals. Circulation 101(23):e215–e220. https://doi.org/10.1161/01.CIR.101.23.e215, http://circ.ahajournals.org/content/101/23/e215

    Article  Google Scholar 

  17. Grimm R, Sukkau J, Hornegger J, Greiner G (2011) Automatic patient pose estimation using pressure sensing mattresses. In: Bildverarbeitung für die Medizin. Springer, pp 409–413

  18. He K, Cao X, Shi Y, Nie D, Gao Y, Shen D (2018) Pelvic organ segmentation using distinctive curve guided fully convolutional networks. IEEE Trans Med Imaging 38(2):585–595

    Article  Google Scholar 

  19. Hu H, Gu J, Zhang Z, Dai J, Wei Y (2018) Relation networks for object detection. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 3588–3597

  20. Imabuchi T, Prima ODA, Ito H (2018) Automated assessment of nonverbal behavior of the patient during conversation with the healthcare worker using a remote camera. eTELEMED 2018. p 29

  21. Insafutdinov E, Pishchulin L, Andres B, Andriluka M, Schiele B (2016) Deepercut: A deeper, stronger, and faster multi-person pose estimation model. In: European conference on computer vision (ECCV). Springer, pp 34–50

  22. Insafutdinov E, Andriluka M, Pishchulin L, Tang S, Levinkov E, Andres B, Schiele B (2017) Arttrack: articulated multi-person tracking in the wild. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 1293–1301

  23. Javaid AQ, Gupta R, Mihalidis A, Etemad SA (2017) Balance-based time-frequency features for discrimination of young and elderly subjects using unsupervised methods. In: IEEE EMBS international conference on biomedical & health informatics (BHI), pp 453–456

  24. Ke L, Chang MC, Qi H, Lyu S (2018) Multi-scale structure-aware network for human pose estimation. In: Proceedings of the european conference on computer vision (ECCV), pp 713–728

  25. Koppula HS, Saxena A (2016) Anticipating human activities using object affordances for reactive robotic response. IEEE Trans Pattern Anal Mach Intell 38(1):14–29

    Article  Google Scholar 

  26. Lee CH, Kim DK, Kim SY, Rhee CS, Won TB (2015) Changes in site of obstruction in obstructive sleep apnea patients according to sleep position: a dise study. Laryngoscope 125(1):248–254

    Article  Google Scholar 

  27. Lee J, Kwon H, Seo J, Shin S, Koo JH, Pang C, Son S, Kim JH, Jang YH, Kim DE et al (2015) Conductive fiber-based ultrasensitive textile pressure sensor for wearable electronics. Adv Mater 27(15):2433–2439

    Article  Google Scholar 

  28. Lin F, Song C, Xu X, Cavuoto L, Xu W (2017) Patient handling activity recognition through pressure-map manifold learning using a footwear sensor. Smart Health 1:77–92

    Article  Google Scholar 

  29. Liu JJ, Huang MC, Xu W, Sarrafzadeh M (2014) Bodypart localization for pressure ulcer prevention. In: Annual international conference of the ieee engineering in medicine and biology society (EMBC), pp 766–769

  30. Liu S, Ostadabbas S (2017) A vision-based system for in-bed posture tracking. In: Proceedings of the IEEE international conference on computer vision workshops, pp 1373–1382

  31. Liu S, Ostadabbas S (2019) Seeing under the cover: A physics guided learning approach for in-bed pose estimation. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 236–245

  32. Liu S, Yin Y, Ostadabbas S (2019) In-bed pose estimation: Deep learning with shallow dataset. IEEE J Transl Eng Health Med 7:1–12

    Article  Google Scholar 

  33. McCabe SJ, Xue Y (2010) Evaluation of sleep position as a potential cause of carpal tunnel syndrome: preferred sleep position on the side is associated with age and gender. Hand 5(4):361–363

    Article  Google Scholar 

  34. McCabe SJ, Gupta A, Tate DE, Myers J (2011) Preferred sleep position on the side is associated with carpal tunnel syndrome. Hand 6(2):132–137

    Article  Google Scholar 

  35. Ostadabbas S, Pouyan MB, Nourani M, Kehtarnavaz N (2014) In-bed posture classification and limb identification. In: IEEE biomedical circuits and systems conference, pp 133–136

  36. Rhodin H, Salzmann M, Fua P (2018) Unsupervised geometry-aware representation for 3d human pose estimation. In: Proceedings of the european conference on computer vision (ECCV), pp 750–767

  37. Ruvalcaba-Cardenas AD, Scoleri T, Day G (2018) Object classification using deep learning on extremely low-resolution time-of-flight data. In: IEEE digital image computing: techniques and applications (DICTA), pp 1–7

  38. Singh V, Ma K, Tamersoy B, Chang YJ, Wimmer A, O’Donnell T, Chen T (2017) Darwin: Deformable patient avatar representation with deep image network. In: International conference on medical image computing and computer-assisted intervention, pp 497–504

  39. Tang W, Yu P, Wu Y (2018) Deeply learned compositional models for human pose estimation. In: European conference on computer vision (ECCV), pp 190–206

  40. Tulyakov S, Liu MY, Yang X, Kautz J (2018) Mocogan: Decomposing motion and content for video generation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1526–1535

  41. Wang CW, Hunter A (2010) Robust pose recognition of the obscured human body. Int J Comput Vis (ICCV) 90(3):313–330

    Article  Google Scholar 

  42. Wei SE, Ramakrishna V, Kanade T, Sheikh Y (2016) Convolutional pose machines. In: IEEE conference on computer vision and pattern recognition, pp 4724–4732

  43. Woo KY, Sears K, Almost J, Wilson R, Whitehead M, VanDenKerkhof EG (2017) Exploration of pressure ulcer and related skin problems across the spectrum of health care settings in ontario using administrative data. Int Wound J 14(1):24–30

    Article  Google Scholar 

  44. Xiao Y, Zijie Z (2020) Infrared image extraction algorithm based on adaptive growth immune field. Neural Process Lett 51(3):2575–2587

    Article  Google Scholar 

  45. Yang W, Li S, Ouyang W, Li H, Wang X (2017) Learning feature pyramids for human pose estimation. In: IEEE international conference on computer vision (ICCV), pp 1290–1299

  46. Zhao ZQ, Zheng P, Xu St, Wu X (2019) Object detection with deep learning: A review. IEEE Trans Neural Netw Learn Syst 30(11):3212–3232

    Article  Google Scholar 

  47. Zhou P, Ni B, Geng C, Hu J, Xu Y (2018) Scale-transferrable object detection. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 528–537

Download references

Acknowledgements

The Titan XP GPU used for this research was donated by the NVIDIA Corporation.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Vandad Davoodnia.

Ethics declarations

Conflict of Interests

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Davoodnia, V., Ghorbani, S. & Etemad, A. Estimating pose from pressure data for smart beds with deep image-based pose estimators. Appl Intell 52, 2119–2133 (2022). https://doi.org/10.1007/s10489-021-02418-y

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-021-02418-y

Keywords

Navigation