Skip to main content
SpringerLink
Log in

BSQ-rate: a New Approach for Video-codec Performance Comparison and Drawbacks of Current Solutions

  • Published:
Programming and Computer Software Aims and scope Submit manuscript

Abstract

This paper is dedicated to the analysis of the existing approaches to video codecs comparisons. It includes the revealed drawbacks of popular comparison methods and proposes new techniques. The performed analysis of user-generated videos collection showed that two of the most popular open video collections from media.xiph.org which are widely used for video-codecs analysis and development do not cover real-life videos complexity distribution. A method for creating representative video sets covering all segments of user videos the spatial and temporal complexity is also proposed. One of the sections discusses video quality estimation algorithms used for video codec comparisons and shows the disadvantages of popular methods VMAF and NIQE. Also, the paper describes the drawbacks of the BD-rate – generally used method for video codecs final ranking during comparisons. A new ranking method called BSQ-rate which considers the identified issues is proposed. The results of this investigation were obtained during the series of research conducted as part of the annual video-codecs comparisons organized by video group of computer graphics and multimedia laboratory at Moscow State University.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.
Fig. 6.
Fig. 7.
Fig. 8.
Fig. 9.
Fig. 10.
Fig. 11.
Fig. 12.
Fig. 13.
Fig. 14.

Similar content being viewed by others

REFERENCES

  1. Cisco VNI Report 2017-2022, 2018 update, 2019. https://www.cisco.com/c/en/us/solutions/collateral/service-provider/visual-networking-index-vni/white-paper-c11-741490.html

  2. Xiph.org Test Media, 2019. https://media.xiph.org/

  3. Video Quality Experts Group Test Sequences, 2019. ftp://ftp.crc.ca/crc/vqeg/TestSequences/

  4. MPEG-2 Transport Stream Test Patterns and Tools, 2019. http://www.w6rz.net/

  5. Sveriges Television: The SVT High Definition Multi-Format Test Set, 2019. ftp://vqeg.its.bldrdoc.gov/HDTV/SVT_MultiFormat/

  6. Columbia Consumer Video (CCV) Database, 2019. http://www.ee.columbia.edu/ln/dvmm/CCV/

  7. CDVL The Consumer Digital Video Library, 2019. https://www.cdvl.org/

  8. LIVE Public-Domain Subjective Video Quality Database, 2019. http://live.ece.utexas.edu/research/quality/live_video.html

  9. Video samples from KODI Wiki, 2019. https://kodi.wiki/view/Samples

  10. Ultra Video Group test sequences, 2019. http://ultravideo.cs.tut.fi/#testsequences

  11. Chen, C., Inguva, S., Rankin, A., and Kokaram, A., A subjective study for the design of multi-resolution ABR video streams with the vp9 codec, Electron. Imag., 2016, vol. 2016, pp. 1–5.

    Google Scholar 

  12. Vatolin, D., Kulikov, D., Erofeev, M., Antsiferova, A., Zvezdakov, S., Kondranin, D., and Grokholsky, S., MSU FullHD Video Codec Comparison 2019. http://compression.ru/video/codec_comparison/hevc_2019/#main_report.

  13. MSU Video Codecs Comparisons, 2019. http://compression.ru/video/codec_comparison/index_en.html

  14. Chikkerur, S., Sundaram, V., Reisslein, M., and Karam, L.J., Objective video quality assessment methods: a classification, review, and performance comparison, IEEE Trans. Broadcast., 2011, vol. 57, no. 2, pp. 165–182.

    Article  Google Scholar 

  15. Wang, Z., Bovik, A.C., Sheikh, H.R., and Simoncelli, E.P., Image quality assessment: from error visibility to structural similarity, IEEE Trans. Image Processing, 2004, vol. 13, no. 4, pp. 600–612.

    Article  Google Scholar 

  16. Sheikh, H.R. and Bovik, A.C., Image information and visual quality, IEEE Trans. Image Process., 2006, vol. 15, no. 2, pp. 430–444.

    Article  Google Scholar 

  17. VMAF: Perceptual video quality assessment based on multi-method fusion, 2019. https://medium.com/netflix-techblog/toward-a-practical-perceptual-video-quality-metric-653f208b9652

  18. MSU Video Codec Comparison 2017. Part III: Full HD Content, Subjective Evaluation. http://www.compression.ru/video/codec_comparison/hevc_2017/MSU_HEVC_comparison_2017_P3_subjective.pdf

  19. Deb, K., Pratap, A., Agarwal, S., and Meyarivan, T.A.M.T., A fast and elitist multiobjective genetic algorithm: NSGA-II, IEEE Trans. Evol. Comput., 2002, vol. 6, no. 2, pp. 182–197.

    Article  Google Scholar 

  20. Vatolin, D.S. and Grishin, S.V., Double up-conversion of video frame rate based on bidirectional motion compensation, Progr. Comput. Software, 2009, vol. 35, no. 6, pp. 351–364.

    Article  Google Scholar 

  21. Mittal, A., Soundararajan, R., and Bovik, A.C., Making a “completely blind” image quality analyzer, IEEE Signal Process. Lett., 2012, vol. 20, no. 3, pp. 209–212.

    Article  Google Scholar 

  22. MSU Video Codec Comparison 2018. Subjective Report. http://compression.ru/video/codec_comparison/hevc_2018/#subjective_report

  23. Bjontegaard, G., Calculation of average PSNR differences between RD-curves, Proc. 13th ITU-T VCEG Meeting, Austin, TX, 2001, document VCEG-M33.

Download references

ACKNOWLEDGMENTS

This work was partially supported by the Russian Foundation for Basic Research under Grant 19-01-00785a. Special thanks to Georgiy Osipov and Denis Kondranin who helped to analyze all detected issues and improved NIQE implementation in MSU VQMT, Mikhail Erofeev who helped with MSU comparison methodology improvement and conducted subjective comparisons, and Moscow State University Graphics and Media Lab team for valuable advice and support in our projects.

Author information

Authors and Affiliations

  1. Moscow State University, Moscow, Russia

    A. V. Zvezdakova, D. L. Kulikov, S. V. Zvezdakov & D. S. Vatolin

  2. Dubna State University, Dubna, Russia

    D. L. Kulikov

Authors
  1. A. V. Zvezdakova
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. D. L. Kulikov
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. S. V. Zvezdakov
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. D. S. Vatolin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to A. V. Zvezdakova, D. L. Kulikov, S. V. Zvezdakov or D. S. Vatolin.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zvezdakova, A.V., Kulikov, D.L., Zvezdakov, S.V. et al. BSQ-rate: a New Approach for Video-codec Performance Comparison and Drawbacks of Current Solutions. Program Comput Soft 46, 183–194 (2020). https://doi.org/10.1134/S0361768820030111

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1134/S0361768820030111

Keywords:

Search

Navigation