Skip to main content
SpringerLink
Log in

An experimental evaluation of visual similarity for HDR images

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In this paper, we investigate visual similarity for high dynamic range (HDR) images. We collect crowdsourcing data through a web-based experimental interface, in which the participants are asked to choose one of the two candidate images as being more similar to the query image. Triplets forming the query-and-candidates sets are obtained by random sampling from existing HDR data sets. Experimental control factors include choice of tone mapping operator (TMO), choice of distance metric, and choice of image feature. The image features that we experiment with are chosen from the features that are commonly used in the usual low dynamic range setting including features learned via Convolutional Neural Networks. The set of image features also includes combined features where the combination coefficients are estimated using logistic regression. We compute correlations between human judgments and quantitative features to understand how much each feature contributes to visual similarity. Combined features yield nearly 84% agreement with human judgments when applied on tone mapped images. Though we observed that using common features directly on raw or linearly scaled HDR images yield subpar correlation estimates compared to using them on tone mapped HDR images, we did not observe significant effect due to the choice of TMO on the estimates. As an application, we propose an improvement to style-based tone mapping for more correctly imparting desired styles to HDR images with different characteristics.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. https://user.ceng.metu.edu.tr/~merve/userstudy/

  2. We unfortunately discovered after the experiments were conducted that one image was duplicated under different names. See the images in 2nd row-4th column and 9th row-3rd column in Fig. 2. In our analysis, we discarded the few trials in which this image was duplicated.

  3. www.microworkers.com

References

  1. Amirkhani D, Bastanfard A (2019) Inpainted image quality evaluation based on saliency map features. In: 2019 5th Iranian Conference on Signal Processing and Intelligent Systems (ICSPIS), pp 1–6

  2. Banterle F, Artusi A, Debattista K, Chalmers A (2011) Advanced high dynamic range imaging: Theory and practice, First. CRC Press (AK Peters), Natick, MA

    Book  Google Scholar 

  3. Bay H, Tuytelaars T, Van Gool L (2006) Surf: Speeded up robust features. In: European conference on computer vision, pp 404–417. Springer

  4. Bhattacharyya A (1946) On a measure of divergence between two multinomial populations. Sankhyō: The Indian Journal of Statistics, pp 401–406

  5. Brown KC, Bryant T, Watkins MD (2010) The forensic application of high dynamic range photography. J Forensic Identification 60(4):449–459

    Google Scholar 

  6. Cai H (2013) High dynamic range photogrammetry for synchronous luminance and geometry measurement. Light Res Technol 45(2):230–257

    Article  Google Scholar 

  7. Chalmers A, Campisi P, Shirley P, Olaizola IG (2016) High dynamic range video: concepts, technologies and applications. Academic Press

  8. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: international Conference on computer vision & Pattern Recognition (CVPR’05), vol 1, pp 886–893. IEEE Computer Society

  9. Debevec PE, Malik J (1997) Recovering high dynamic range radiance maps from photographs. In: SIGGRAPH 97 Conference Proceedings, pp 369–378

  10. Donahue J, Jia Y, Vinyals O, Hoffman J, Zhang N, Tzeng E, Darrell T (2014) Decaf: A deep convolutional activation feature for generic visual recognition. In: International conference on machine learning, pp 647–655

  11. Drago F, Myszkowski K, Annen T, Chiba N (2003) Adaptive logarithmic mapping for displaying high contrast scenes. In: Computer Graphics Forum, vol 22, pp 419–426. Wiley Online Library

  12. Durand F, Dorsey J (2002) Fast bilateral filtering for the display of high-dynamic-range images. ACM Trans Graph 21(3):257–266

    Article  Google Scholar 

  13. Empa hdr image database. http://www.empamedia.ethz.ch/hdrdatabase/ Accessed: 2017-08-26

  14. Fairchild M D (2007) The hdr photographic survey. In: Color and Imaging Conference, pp 233–238. Society for Imaging Science and Technology

  15. Fattal R, Lischinski D, Werman M (2002) Gradient domain high dynamic range compression. ACM Trans Graph 21(3):249–256

    Article  Google Scholar 

  16. Ferradans S, Bertalmio M, Provenzi E, Caselles V (2011) An analysis of visual adaptation and contrast perception for tone mapping. IEEE Transactions on Pattern Analysis and Machine Intelligence 33(10):2002–2012

    Article  Google Scholar 

  17. Ferwerda JA, Pattanaik S, Shirley P, Greenberg DP (1996) A model of visual adaptation for realistic image synthesis. In: SIGGRAPH 96 Conference Proceedings, pp 249–258

  18. Frese T, Bouman CA, Allebach JP (1997) Methodology for designing image similarity metrics based on human visual system models. In: Human Vision and Electronic Imaging II, vol 3016, pp 472–483. International Society for Optics and Photonics

  19. Froehlich J, Grandinetti S, Eberhardt B, Walter S, Schilling A, Brendel H (2014) Creating cinematic wide gamut hdr-video for the evaluation of tone mapping operators and hdr-displays. In: Digital Photography X, vol 9023, p 90230X. International Society for Optics and Photonics

  20. Glassner AS (1995) Principles of digital image synthesis: Vol. 1, Elsevier

  21. Gordo A, Almazán J, Revaud J, Larlus D (2016) Deep image retrieval: Learning global representations for image search. In: European conference on computer vision, pp 241–257. Springer

  22. Grimaldi A, Kane D, Bertalmío M (2019) Statistics of natural images as a function of dynamic range. J Vis 19(2):13–13. https://doi.org/10.1167/19.2.13

    Article  Google Scholar 

  23. Grinzato E, Cadelano G, Bison P, Petracca A (2009) Seismic risk evaluation aided by ir thermography. In: SPIE Defense, Security, and Sensing, pp 72990C–72990C. International Society for Optics and Photonics

  24. Hanhart P, Bernardo MV, Pereira M, Pinheiro AMG, Ebrahimi T (2015) Benchmarking of objective quality metrics for hdr image quality assessment. EURASIP Journal on Image and Video Processing 2015(1):1–18

    Article  Google Scholar 

  25. Happa J, Artusi A, Czanner S, Chalmers A (2010) High dynamic range video for cultural heritage documentation and experimental archaeology. In: Proceedings of the 11th International conference on Virtual Reality, Archaeology and Cultural Heritage, pp 17–24. Eurographics Association

  26. Harifi S, Bastanfard A (2015) Efficient iris segmentation based on converting iris images to high dynamic range images. In: 2015 Second International Conference on Computing Technology and Information Management (ICCTIM), pp 115–119. IEEE

  27. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  28. ISO EN (2011) 11664-4 colorimetry—part 4: Cie 1976 l* a* b* colour space. CEN (European Committee for Standardization): Brussels, Belgium

  29. Kalantari NK, Ramamoorthi R (2017) Deep high dynamic range imaging of dynamic scenes. ACM Trans. Graph 36(4):144

    Article  Google Scholar 

  30. Kleiman Y, Goldberg G, Amsterdamer Y, Cohen-Or D (2016) Toward semantic image similarity from crowdsourced clustering. Vis Comput 32 (6-8):1045–1055

    Article  Google Scholar 

  31. Klíma M, Fliegel K, Pata P, Vitek S, Blažek M, Dostal P, Krasula L, Kratochvíl T, Rícnỳ V, Slanina M et al (2011) Deimos–an open source image database. Radioengineering, vol 20 (4)

  32. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  33. Kundu D, Ghadiyaram D, Bovik AC, Evans BL (2016) Espl-live hdr image quality database. Online: http://signal.ece.utexas.edu/debarati/HDRDatabase.zip,[Mar, 2017]

  34. Kundu D, Ghadiyaram D, Bovik AC, Evans BL (2017) Large-scale crowdsourced study for high dynamic range images. IEEE Trans Image Process 26(10):4725–4740

    Article  MathSciNet  Google Scholar 

  35. Larson GW, Shakespeare RA (1998) Rendering with radiance. Morgan Kaufmann Publishers

  36. Liu Y, Zhang D, Lu G, Ma W-Y (2007) A survey of content-based image retrieval with high-level semantics. Pattern recognition 40(1):262–282

    Article  Google Scholar 

  37. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. International journal of computer vision 60(2):91–110

    Article  Google Scholar 

  38. Lun Z, Kalogerakis E, Sheffer A (2015) Elements of style: learning perceptual shape style similarity. ACM Transactions on Graphics (TOG) 34(4):84

    Article  Google Scholar 

  39. Mai Z, Mansour H, Mantiuk R, Nasiopoulos P, Ward R, Heidrich W (2011) Optimizing a tone curve for backward-compatible high dynamic range image and video compression. IEEE Trans Image Process 20(6):1558–1571. https://doi.org/10.1109/TIP.2010.2095866

    Article  MathSciNet  MATH  Google Scholar 

  40. Mantiuk R (2007) High dynamic range imaging: towards the limits of the human visual perception. Forsch. Wiss. Rechnen 72:11–27

    Google Scholar 

  41. Mantiuk R, Daly S, Kerofsky L (2008) Display adaptive tone mapping. ACM Trans. Graph. 27:68:1–68:10. https://doi.org/10.1145/1360612.1360667

    Article  Google Scholar 

  42. Mantiuk R, Heidrich W (2009) Visualizing high dynamic range images in a web browser. J Graphics, GPU, and Game Tools 14(1):43–53

    Article  Google Scholar 

  43. Mantiuk R, Kim KJ, Rempel AG, Heidrich W (2011) Hdr-vdp-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions. ACM Trans. Graph. 30(4):40:1–40:14. https://doi.org/10.1145/2010324.1964935

    Article  Google Scholar 

  44. Mantiuk R, Myszkowski K, Seidel H-P (2006) A perceptual framework for contrast processing of high dynamic range images. ACM Transactions on Applied Perception (TAP) 3(3):286–308

    Article  Google Scholar 

  45. Mantiuk R, Seidel H-P (2008) Modeling a generic tone-mapping operator. Computer Graphics Forum 27(2):699–708

    Article  Google Scholar 

  46. Narwaria M, Da Silva MP, Le Callet P (2015) Hdr-vqm: An objective quality measure for high dynamic range video. Signal Process Image Commun 35:46–60

    Article  Google Scholar 

  47. Nemoto H, Korshunov P, Hanhart P, Ebrahimi T (2015) Visual attention in ldr and hdr images. In: 9th International Workshop on Video Processing and Quality Metrics for Consumer Electronics (VPQM)

  48. Neumann D, Gegenfurtner KR (2006) Image retrieval and perceptual similarity. ACM Transactions on Applied Perception (TAP) 3(1):31–47

    Article  Google Scholar 

  49. Oğuz Akyüz A, Bloch MAC, Hadimli K (2013) Style-based tone mapping for hdr images. In: SIGGRAPH Asia 2013 Technical Briefs. ACM. No. 39

  50. Oliva A, Torralba A (2001) Modeling the shape of the scene: A holistic representation of the spatial envelope. International journal of computer vision 42(3):145–175

    Article  Google Scholar 

  51. Parraga CA, Otazu X, et al. (2018) Which tone-mapping operator is the best? a comparative study of perceptual quality. JOSA A 35(4):626–638

    Article  Google Scholar 

  52. Pattanaik SN, Tumblin J, Yee H, Greenberg DP (2000) Time-dependent visual adaptation for fast realistic image display. In: Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pp 47–54. ACM Press/Addison-Wesley Publishing Co.

  53. Rawat S, Gairola S, Shah R, Narayanan PJ (2018) Find me a sky: A data-driven method for color-consistent sky search and replacement. In: International Conference on Multimedia Modeling, pp 216–228. Springer

  54. Reinhard E, Devlin K (2005) Dynamic range reduction inspired by photoreceptor physiology. IEEE Trans Vis Comput Graph 11(1):13–24

    Article  Google Scholar 

  55. Reinhard E, Stark M, Shirley P, Ferwerda J (2002) Photographic tone reproduction for digital images. ACM Trans Graph 21(3):267–276

    Article  Google Scholar 

  56. Reinhard E, Ward G, Pattanaik S, Debevec P (2010) High dynamic range imaging: Acquisition, display and image-based lighting, Second. Morgan Kaufmann, San Francisco

    Google Scholar 

  57. Rizzi A, Barricelli BR, Bonanomi C, Albani L, Gianini G (2018) Visual glare limits of hdr displays in medical imaging. IET Comput Vis 12(7):976–988

    Article  Google Scholar 

  58. Rogowitz BE, Frese T, Smith JR, Bouman CA, Kalin EB (1998) Perceptual image similarity experiments. In: Photonics West’98 Electronic Imaging, pp 576–590

  59. Rubner Y, Tomasi C, Guibas LJ (2000) The earth mover’s distance as a metric for image retrieval. International journal of computer vision 40(2):99–121

    Article  Google Scholar 

  60. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M et al (2015) Imagenet large scale visual recognition challenge. International journal of computer vision 115 (3):211–252

    Article  MathSciNet  Google Scholar 

  61. STANDARD SMPTE (2016) Dynamic metadata for color volume transform–core components

  62. Saleh B, Dontcheva M, Hertzmann A, Liu Z (2015) Learning style similarity for searching infographics. In: Proceedings of the 41st graphics interface conference, pp 59–64. Canadian Information Processing Society

  63. Seetzen H, Heidrich W, Stuerzlinger W, Ward G, Whitehead L, Trentacoste M, Ghosh A, Vorozcovs A (2004) High dynamic range display systems. ACM Trans Graph 23(3):760–768

    Article  Google Scholar 

  64. Sen P, Kalantari NK, Yaesoubi M, Darabi S, Goldman DB, Shechtman E (2012) Robust patch-based hdr reconstruction of dynamic scenes. ACM Trans. Graph. 31(6):203

    Article  Google Scholar 

  65. Sharma M, Ghosh H (2015) Histogram of gradient magnitudes: a rotation invariant texture-descriptor. In: 2015 IEEE International Conference on Image Processing (ICIP), pp 4614–4618. IEEE

  66. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556

  67. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  68. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9

  69. Theodor JM, Furr RS (2009) High dynamic range imaging as applied to paleontological specimen photography. Palaeontol Electron, 12(1)

  70. Tocci MD, Kiser C, Tocci N, Sen P (2011) A versatile HDR video production system. In: ACM Transactions on Graphics (TOG), 30, p 41. ACM

  71. Tumblin J, Rushmeier H (1993) Tone reproduction for computer generated images. IEEE Comput Graph Appl 13(6):42–48

    Article  Google Scholar 

  72. Upton GJG (1992) Fisher’s exact test. J Royal Statistical Society: Series A (Statistics in Society) 155(3):395–402

    Article  Google Scholar 

  73. Wan J, Wang D, Hoi SCH, Wu P, Zhu J, Zhang Y, Li J (2014) Deep learning for content-based image retrieval: A comprehensive study. In: Proceedings of the 22nd ACM international conference on Multimedia, pp 157–166

  74. Ward G, Rushmeier H, Piatko C (1997) A visibility matching tone reproduction operator for high dynamic range scenes. IEEE Trans. on Visualization and Comp. Graphics, 3(4)

  75. Wu H-HP, Lee Y-P, Chang S-H (2012) Fast measurement of automotive headlamps based on high dynamic range imaging. Applied optics 51 (28):6870–6880

    Article  Google Scholar 

  76. Yeganeh H, Wang Z (2012) Objective quality assessment of tone-mapped images. IEEE Transactions on Image processing 22(2):657–667

    Article  MathSciNet  Google Scholar 

  77. Yosinski J, Clune J, Bengio Y, Lipson H (2014) How transferable are features in deep neural networks?. In: Advances in neural information processing systems, pp 3320–3328

  78. Zhang B, Srihari SN (2003) Properties of binary vector dissimilarity measures. In: Proc. JCIS Int’l Conf. Computer Vision, Pattern Recognition, and Image Processing, 1. Citeseer

  79. Zhou B, Zhao H, Puig X, Fidler S, Barriuso A, Torralba A (2017) Scene parsing through ade20k dataset. In: Proc. CVPR

Download references

Author information

Authors and Affiliations

  1. Middle East Technical University, Department of Computer Engineering, Ankara, Turkey

    Merve Aydinlilar, Ahmet Oguz Akyuz & Sibel Tari

Authors
  1. Merve Aydinlilar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ahmet Oguz Akyuz
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sibel Tari
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ahmet Oguz Akyuz.

Ethics declarations

Conflict of Interests

All authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Aydinlilar, M., Akyuz, A.O. & Tari, S. An experimental evaluation of visual similarity for HDR images. Multimed Tools Appl 80, 32449–32472 (2021). https://doi.org/10.1007/s11042-021-11182-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-11182-7

Keywords

Search

Navigation