Skip to main content
SpringerLink
Log in

Psycho-visual modulation based information display: introduction and survey

  • Review Article
  • Published:
Frontiers of Computer Science Aims and scope Submit manuscript

Abstract

Industry and academia have been making great efforts in improving refresh rates and resolutions of display devices to meet the ever increasing needs of consumers for better visual quality. As a result, many modern displays have spatial and temporal resolutions far beyond the discern capability of human visual systems. Thus, leading to the possibility of using those display-eye redundancies for innovative usages. Temporal/spatial psycho-visual modulation (TPVM/SPVM) was proposed to exploit those redundancies to generate multiple visual percepts for different viewers or to transmit non-visual data to computing devices without affecting normal viewing. This paper reviews the STPVM technology from both conceptual and algorithmic perspectives, with exemplary applications in multi-view display, display with visible light communication, etc. Some possible future research directions are also identified.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Wu X, Zhai G. Temporal psychovisual modulation: a new paradigm of information display [exploratory DSP]. IEEE Signal Processing Magazine, 2013, 30(1): 136–141

    Article  Google Scholar 

  2. Kelly D H. Spatio-temporal frequency characteristics of color-vision mechanisms. Journal of the Optical Society America, 1974, 64(7): 983–990

    Article  Google Scholar 

  3. Varner D, Jameson D, Hurvich L M. Temporal sensitivities related to color theory. Journal of the Optical Society of America A, 1984, 1(5): 474–481

    Article  Google Scholar 

  4. Instruments T. DLP discovery 4100 development kit. See Ti Website, 2015

  5. Corporation N. NVIDIA 3D Vision. See Nvidia Website, 2014

  6. Ko H, Paik J, Zalewski G. Stereoscopic screen sharing method and apparatus. 2010. US Patent App. 12/503,029

  7. Corporation N. Pixel density display listing. See Pixensity Website, 2018

  8. Karnik A, Martinez Plasencia D, Mayol-Cuevas W, Subramanian S. PiVOT: personalized view-overlays for tabletops. In: Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology. 2012, 271–280

  9. Wetzstein G, Lanman D, Hirsch M, Raskar R. Tensor displays: compressive light field synthesis using multilayer displays with directional backlighting. ACM Transactions on Graphics, 2012, 31(4): 80

    Article  Google Scholar 

  10. Ye G, State A, Fuchs H. A practical multi-viewer tabletop autostereoscopic display. In: Proceedings of IEEE International Symposium on Mixed and Augmented Reality. 2010, 147–156

  11. Karnik A, Mayol-Cuevas W, Subramanian S. MUSTARD: a multi user see through AR display. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2012, 2541–2550

  12. Nashel A, Fuchs H. Random hole display: a non-uniform barrier autostereoscopic display. In: Proceedings of 3DTV Conference: The True Vision — Capture, Transmission and Display of 3D Video. 2009, 1–4

  13. Lanman D, Wetzstein G, Hirsch M, Heidrich W, Raskar R. Polarization fields: dynamic light field display using multi-layer LCDs. In: Proceedings of the SIGGRAPH Asia Conference. 2011, 1–10

  14. Zhai G, Wu X. Multiuser collaborative viewport via temporal psychovisual modulation [applications corner]. IEEE Signal Processing Magazine, 2014, 31(5): 144–149

    Article  Google Scholar 

  15. Wu X, Zhai G. Backward compatible stereoscopic displays via temporal psychovisual modulation. In: Proceedings of SIGGRAPH Asia Emerging Technologies. 2012

  16. Jiao L, Shu X, Wu X. LED backlight adjustment for backward-compatible stereoscopic display. IEEE Signal Processing Letters, 2013, 20(12): 1203–1206

    Article  Google Scholar 

  17. Ma R, Au O C, Wan P, Xu L, Sun W, Hu W. Improved temporal psychovisual modulation for backward-compatible stereoscopic display. In: Proceedings of IEEE Global Conference on Signal and Information Processing. 2014, 1034–1038

  18. Chen Y, Zhai G, Zhou J, Wan Z, Tang L. Global quality of assessment and optimization for the backward-compatible stereoscopic display system. In: Proceedings of IEEE International Conference on Image Processing. 2017, 191–195

  19. Fujimura W, Koide Y, Songer R, Hayakawa T, Shirai A, Yanaka K. 2x3D: real time shader for simultaneous 2D/3D hybrid theater. In: Proceedings of SIGGRAPH Asia Emerging Technologies. 2012, 1–2

  20. Scher S, Liu J, Vaish R, Gunawardane P, Davis J. 3D+2DTV: 3D displays with no ghosting for viewers without glasses. ACM Transactions on Graphics, 2013, 32(3): 21

    Article  Google Scholar 

  21. Gao Z, Zhai G, Min X. Information security display system based on temporal psychovisual modulation. In: Proceedings of IEEE International Symposium on Circuits and Systems. 2014, 449–452

  22. Hu C, Zhai G, Gao Z, Min X. Information security display system based on spatial psychovisual modulation. In: Proceedings of IEEE International Conference on Multimedia and Expo. 2014, 1–4

  23. Chen Y, Liu N, Zhai G, Gao Z, Gu K. Information security display system on android device. In: Proceedings of IEEE Region 10 Conference. 2016, 1634–1637

  24. Li X, Zhai G, Wang J, Gu K. Portable information security display system via spatial psychovisual modulation. In: Proceedings of IEEE Visual Communications and Image Processing. 2017, 1–4

  25. Hu C, Zhai G, Gao Z, Min X. Simultaneous dual-subtitles exhibition via spatial psychovisual modulation. In: Proceedings of IEEE International Symposium on Broadband Multimedia Systems and Broadcasting. 2014, 1–4

  26. Hu C, Zhai G, Gao Z, Min X. Simultaneous triple subtitles exhibition via temporal psychovisual modulation. In: Proceedings of the 9th IEEE Conference on Industrial Electronics and Applications. 2014, 944–947

  27. Sun W, Zhai G, Gao Z, Chen T, Zhu Y, Wang Z. Dual-view oracle bone script recognition system via temporal-spatial psychovisual modulation. In: Proceedings of IEEE Conference on Multimedia Information Processing and Retrieval. 2020, 193–198

  28. Gao Z, Zhai G, Hu C, Min X. Dual-view medical image visualization based on spatial-temporal psychovisual modulation. In: Proceedings of IEEE International Conference on Image Processing. 2014

  29. Fang W, Zhai G, Yang X, Liu J, Chen Y. An eye-friendly dual-view projection system using temporal psychovisual modulation. In: Proceedings of IEEE International Symposium on Broadband Multimedia Systems and Broadcasting. 2017, 1–5

  30. Zhai G, Wu X. Defeating camcorder piracy by temporal psychovisual modulation. Journal of Display Technology, 2014, 10(9): 754–757

    Article  Google Scholar 

  31. Gao Z, Zhai G, Wu X, Min X, Zhi C. DLP based anti-piracy display system. In: Proceedings of IEEE Conference on Visual Communications and Image Processing. 2014, 145–148

  32. Chen Y, Zhai G, Gao Z, Gu K, Zhang W, Hu M, Liu J. Movie piracy tracking using temporal psychovisual modulation. In: Proceedings of IEEE International Symposium on Broadband Multimedia Systems and Broadcasting. 2017, 1–4

  33. Gao Z, Zhai G, Hu C. The invisible QR code. In: Proceedings of the ACM International Conference on Multimedia. 2015, 675–678

  34. Lu X, You B, Lin P Y. Augmented reality via temporal psycho-visual modulation. In: Proceedings of IEEE International Conference on Multimedia Expo Workshops. 2016, 1–4

  35. Chen Q, Chen Y. Polarization based invisible barcode display. In: Proceedings of International Forum on Digital TV and Wireless Multimedia Communication. 2018, 67–77

  36. Shi S, Chen L, Hu W, Gruteser M. Reading between lines: high-rate, non-intrusive visual codes within regular videos via implicitcode. In: Proceedings of the ACM International Joint Conference on Pervasive and Ubiquitous Computing. 2015, 157–168

  37. Wu X, Shu X. Combining information display and visible light wireless communication. In: Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing. 2015, 1657–1661

  38. Shu X, Wu X. Frame untangling for unobtrusive display-camera visible light communication. In: Proceedings of the 24th ACM International Conference on Multimedia. 2016, 650–654

  39. Liu K, Wu X, Shu X. On display-camera synchronization for visible light communication. In: Proceedings of Visual Communications and Image Processing. 2015, 1–4

  40. Hu C, Zhai G, Gao Z. Visible light communication via temporal psychovisual modulation. In: Proceedings of the 23rd ACM International Conference on Multimedia. 2015, 785–788

  41. Fang W, Zhai G, Yang X. A flash light system for individuals with visual impairment based on TPVM. In: Proceedings of International Conference on Cloud Computing and Big Data. 2016, 362–366

  42. Zhang Y, Zhai G, Liu J, Weng X, Chen Y. ‘window of visibility’ inspired security lighting system. In: Proceedings of International Conference on Systems, Signals and Image Processing. 2017, 1–5

  43. Gao Z, Zhai G, Zhou J. Factorization algorithms for temporal psychovisual modulation display. IEEE Transactions on Multimedia, 2016, 18(4): 614–626

    Article  Google Scholar 

  44. Feng J, Huo X, Song L, Yang X, Zhang W. Evaluation of different algorithms of nonnegative matrix factorization in temporal psychovisual modulation. IEEE Transactions on Circuits and Systems for Video Technology, 2014, 24(4): 553–565

    Article  Google Scholar 

  45. Wang L, Zhai G. Constrained nmf for multiple exhibition on a single display. In: Proceedings of Picture Coding Symposium. 2015, 292–296

  46. Gao Z, Zhai G, Gu X, Zhou J. Adapting hierarchical ALS algorithms for temporal psychovisual modulation. In: Proceedings of IEEE International Symposium on Circuits and Systems. 2015, 2756–2759

  47. Gao Z, Zhai G, Wang J. Spatially-weighted nonnegative matrix factorization with application to temporal psychovisual modulation. Digital Signal Processing, 2017, 67: 123–130

    Article  Google Scholar 

  48. Kim J, Park H. Fast nonnegative matrix factorization: an active-set-like method and comparisons. SIAM Journal on Scientific Computing, 2011, 33(6): 3261–3281

    Article  MathSciNet  Google Scholar 

  49. Lee D D, Seung H S. Learning the parts of objects by non-negative matrix factorization. Nature, 1999, 401(6755): 788–791

    Article  Google Scholar 

  50. Gonzalez E F, Zhang Y. Accelerating the Lee-Seung algorithm for nonnegative matrix factorization. Department of Computational and Applied Mathematics, Rice University, Houston, TX, Technical Report, TR-05-02, 2005

  51. Berry M W, Browne M, Langville A N, Pauca V P, Plemmons R J. Algorithms and applications for approximate nonnegative matrix factorization. Computational Statistics & Data Analysis, 2007, 52(1): 155–173

    Article  MathSciNet  Google Scholar 

  52. Kim J, Park H. Toward faster nonnegative matrix factorization: a new algorithm and comparisons. In: Proceedings of IEEE International Conference on Data Mining. 2008, 353–362

  53. Kim H, Park H. Nonnegative matrix factorization based on alternating nonnegativity constrained least squares and active set method. SIAM Journal on Matrix Analysis and Applications, 2008, 30(2): 713–730

    Article  MathSciNet  Google Scholar 

  54. Cichocki A, Anh-Huy P. Fast local algorithms for large scale nonnegative matrix and tensor factorizations. IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, 2009, 92(3): 708–721

    Article  Google Scholar 

  55. Gillis N, Glineur F. Accelerated multiplicative updates and hierarchical als algorithms for nonnegative matrix factorization. Neural Computation, 2012, 24(4): 1085–1105

    Article  MathSciNet  Google Scholar 

  56. Itti L, Koch C, Niebur E. A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1998, 20(11): 1254–1259

    Article  Google Scholar 

  57. Reinhard E, Ward G, Pattanaik S, Debevec P. High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting (The Morgan Kaufmann Series in Computer Graphics). San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 2005

    Google Scholar 

  58. Li D, Gao Z, Zhang X P, Zhai G, Yang X. Generative adversarial networks for non-negative matrix factorization in temporal psycho-visual modulation. Digital Signal Processing, 2020, 100: 102681

    Article  Google Scholar 

  59. Gao Z, Zhai G. Dual-view display based on spatial psychovisual modulation. IEEE Access, 2018, 6: 41356–41366

    Article  Google Scholar 

  60. NÑsÑnen R E, Kukkonen H T, Rovamo J M. A window model for spatial integration in human pattern discrimination. Investigative Ophthalmology & Visual Science, 1995, 36(9): 1855–1862

    Google Scholar 

  61. Watamaniuk S N J, Sekuler R. Temporal and spatial integration in dynamic random-dot stimuli. Vision Research, 1992, 32(12): 2341–2347

    Article  Google Scholar 

  62. Sun W, Gao Z, Zhai G, Zhang J, Wang Z, Zhu Y. An improved algorithm for real-time dual-view display. In: Proceedings of IEEE International Symposium on Circuits and Systems. 2020, 1–5

  63. Zhai G, Min X. Perceptual image quality assessment: a survey. Science China Information Sciences, 2020, 63(11): 211301

    Article  Google Scholar 

  64. Chen Y, Liu N, Zhai G, Gu K, Wang J, Gao Z, Zhu Y. Quality assessment for dual-view display system. In: Proceedings of Visual Communications and Image Processing. 2016, 1–4

  65. Chen Y, Zhai G, Gu K, Zhang X, Lin W, Zhou J. Benchmarking screen content image quality evaluation in spatial psychovisual modulation display system. In: Proceedings of Pacific Rim Conference on Multimedia. 2018, 629–640

  66. Scherzinger A L, Hendee W R. Basic principles of magnetic resonance imaging-an update. The Western Journal of Medicine, 1985, 143(6): 782–792

    Google Scholar 

  67. Bushong S C, Clarke G. Magnetic Resonance Imaging: Physical and Biological Principles. Elsevier Health Sciences, 2014

  68. Lambooij M, IJsselsteijn W, Heynderickx I. Visual discomfort of 3d tv: assessment methods and modeling. Displays, 2011, 32(4): 209–218

    Article  Google Scholar 

  69. Taylor S A. CCD and CMOS imaging array technologies: technology review. Xerox Research Centre Europe,Technical Report EPC-1998-106, 1998, 1–14

  70. ISO/IEC. Information technology-automatic identification and data capture techniques-QR Code 2005 bar code symbology specification. see Iso.org Website, 2006

  71. Song K, Liu N, Gao Z, Zhang J, Zhai G, Zhang X P. Deep restoration of invisible QR code from TPVM display. In: Proceedings of IEEE International Conference on Multimedia & Expo Workshops. 2020, 1–6

  72. Siwek S E. The true cost of copyright industry piracy to the US economy. IPI Center for Technology Freedom, 2007

  73. Dorning J. Intellectual Property Theft: A Threat to U.S. Workers, Industries, and Our Economy. DPE Research Department, 2014

  74. NEWS B. The fact and fiction of camcorder piracy. see Bbc.co.uk Website, 2015

  75. Byers S, Cranor L F, Cronin E, Korman D, McDaniel P. An analysis of security vulnerabilities in the movie production and distribution process. Telecommunications Policy, 2004, 28(78): 619–644

    Article  Google Scholar 

  76. Haitsma J, Kalker T. A watermarking scheme for digital cinema. In: Proceedings of International Conference on Image Processing. 2001, 487–489

  77. Nguyen P, Balter R, Montfort N, Baudry S. Registration methods for nonblind watermark detection in digital cinema applications. In: Proceedings of Security and Watermarking of Multimedia Contents V. 2003, 553–562

  78. Lubin J, Bloom J A, Cheng H. Robust content-dependent high-fidelity watermark for tracking in digital cinema. In: Proceedings of Security and Watermarking of Multimedia Contents V. 2003, 536–545

  79. Nakashima Y, Tachibana R, Babaguchi N. Watermarked movie soundtrack finds the position of the camcorder in a theater. IEEE Transactions on Multimedia, 2009, 11(3): 443–454

    Article  Google Scholar 

  80. Davis J, Hsieh Y H, Lee H C. Humans perceive flicker artifacts at 500 hz. Scientific Reports, 2015, 5: 7861

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like thanks the National Natural Science Foundation of China (NSFC) for the support (Grant Nos. 61901259, 61831015, 61771305, 61927809, and U1908210) and China Postdoctoral Science Foundation (BX2019208).

Author information

Author notes

  1. These authors contributed equally to this work

Authors and Affiliations

  1. Cooperative Medianet Innovation Center, Shanghai Jiao Tong University, Shanghai, 200240, China

    Ning Liu

  2. MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, Shanghai, 200240, China

    Zhongpai Gao

  3. Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University, Shanghai, 200240, China

    Jia Wang & Guangtao Zhai

Authors
  1. Ning Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhongpai Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jia Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Guangtao Zhai
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guangtao Zhai.

Additional information

Ning Liu received the PhD degree from Shanghai Jiao Tong University, China in 2010. He is currently an assistant professor with Shanghai Jiao Tong University, China. His research interests include wireless and mobile communication systems, net-work media information security, intelligent hard-ware, and mobile Internet.

Zhongpai Gao received the BSc degree in electronic and information engineering from the Huazhong University of Science and Technology, China in 2013 and the PhD degree from Shanghai Jiao Tong University, China in 2018. He was a visiting PhD Student with the Schepens Eye Research Institute, Harvard Medical School, USA, from 2016 to 2018. Currently, he is a postdoctoral fellow in Shanghai Jiao Tong University, China. His research interests include stereoscopic 3D and 3D computer vision.

Jia Wang received the BSc degree in electronic engineering, the MS degree in pattern recognition and intelligence control, and the PhD degree in electronic engineering from Shanghai Jiao Tong University, China in 1997, 1999, and 2002, respectively. He is currently a professor with the Department of Electronic Engineering, Shanghai Jiao Tong University, and also a member of the Shanghai Key Laboratory of Digital Media Processing and Transmission. His research interests include multiuser information theory and mathematics for artificial intelligence.

Guangtao Zhai (Senior Member, IEEE) received the BE and ME degrees from Shandong University, China in 2001 and 2004, respectively, and the PhD degree from Shanghai Jiao Tong University, China in 2009. From 2008 to 2009, he was a Visiting Student with the Department of Electrical and Computer Engineering, McMaster University, Canada, where he was a post-doctoral fellow from 2010 to 2012. From 2012 to 2013, he was a Humboldt Research Fellow with the Institute of Multimedia Communication and Signal Processing, Friedrich Alexander University of Erlangen-Nuremberg, Germany. He is currently a Research Professor with the Institute of Image Communication and Information Processing, Shanghai Jiao Tong University. His research interests include multimedia signal processing and perceptual signal processing. He received the Award of National Excellent PhD Thesis from the Ministry of Education of China in 2012.

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, N., Gao, Z., Wang, J. et al. Psycho-visual modulation based information display: introduction and survey. Front. Comput. Sci. 15, 153703 (2021). https://doi.org/10.1007/s11704-019-8265-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11704-019-8265-3

Keywords

Search

Navigation