VQMT3D Project: Report 10 on 3D-video Quality Analysis

In cooperation with IITP RAS

Blu-ray discs purchased for testing

About the Report

This is the 10th Anniversary Report of the VQMT3D project. It contains a detailed analysis of 105 stereoscopic (S3D) movies. It has some significant differences from previous reports released by us.

We are publishing this report almost four years after its creation due to several reasons. We’ve cooperated with multiple companies and organizations, but their support in recent years has dwindled to zero. Moreover, although several companies whose business depends heavily on the success of 3D movies promised to aid our efforts, these promises went unfulfilled. We’re nevertheless happy to finally present you with our 10th Anniversary Report.

Over 100 films analysed, including:

Examined problems:

  • Gravity (2013)
  • The Great Gatsby (2013)
  • Oz the Great and Powerful (2013)
  • Stalingrad (2013)
  • Iron Man 3 (2013)
  • Man of Steel (2013)
  • Pacific Rim (2013)
  • Jack the Giant Slayer (2013)
  • The Hobbit: An Unexpected Journey (2012)
  • The Legend of Hercules (2014)
  • Alice in Wonderland (2010)
  • Pirates of the Caribbean: On Stranger Tides (2011)
  • The Avengers (2012)
  • The Amazing Spiderman (2012)
  • World War Z (2013)
  • Thor (2011)
  • The Last Airbender (2010)
  • Life of Pi (2012)
  • TRON: Legacy (2010)
  • Hugo (2011)
  • And more...
  • Depth Budget
  • Depth Continuity
  • Vertical Parallax
  • Scale and Rotation Mismatch
  • Color Mismatch
  • Sharpness Mismatch
  • Stereo Window Violation
  • Temporal Shift
  • Channel Mismatch
  • Crosstalk
  • Average brightness

Report contents and diagram examples

Firstly, we present an overall comparison of the movies we evaluated. It includes charts depicting the average metric values relative to a movie’s release date and budget, as well as charts illustrating metric-value distributions.

Overall diagram
An example diagram which compares the depth continuity metric value and budget for each movie. Higher values indicate more frequent and intense depth jump cuts throughout a film.
Bar chart diagram
Color-mismatch values represent the strength of color difference between two views of a stereoscopic frame. An example bar-chart which shows average color-mismatch for each movie.

In the next section we rate movies on the basis of our metrics for multiple categories. Different metrics correspond to different categories, as do low- versus high-budget movies and movies with distinct release years. We also present an overall technical-quality comparison that combines all the relevant results for the various quality metrics.

Top 10 movies by stereo window placement comfort
An example overall movie rating with top 10 best movies in terms of stereo window placement comfort.
Top 10 technical quality movies
This example diagram shows the technical quality category of low-budget movies, which is based on average movie rating by each metric.

We sum up the ratings in the previous section by concisely listing movies by their total number of nominations.

The final part describes our plans for continuing this project.

Publications

If you want to make a reference to this project, please refer to one of the following publications:

  1. Mikhail Erofeev, Dmitriy Vatolin, Alexander Voronov, Alexey Fedorov,
    “Toward an Objective Stereo-Video Quality Metric: Depth Perception of Textured Areas,”
    International Conference on 3D Imaging,
    2012. doi:10.1109/IC3D.2012.6615120 (download)
  2. Dmitriy Akimov, Alexey Shestov, Alexander Voronov, Dmitriy Vatolin,
    “Automatic Left-Right Channel Swap Detection,”
    International Conference on 3D Imaging,
    2012. doi:10.1109/IC3D.2012.6615126 (download)
  3. Alexander Voronov, Alexey Borisov, Dmitriy Vatolin,
    “System for automatic detection of distorted scenes in stereo video,”
    International Workshop on Video Processing and Quality Metrics for Consumer Electronic (VPQM-2012),
    pp. 138–143, 2012. (download)
  4. Alexander Voronov, Dmitriy Vatolin, Denis Sumin, Vyacheslav Napadovsky, Alexey Borisov,
    “Towards Automatic Stereo-video Quality Assessment and Detection of Color and Sharpness Mismatch,”
    International Conference on 3D Imaging,
    2012. doi:10.1109/IC3D.2012.6615121 (download)
  5. Alexander Voronov, Dmitriy Vatolin, Denis Sumin, Vyacheslav Napadovsky, Alexey Borisov,
    “Methodology for stereoscopic motion-picture quality assessment,”
    Proc. SPIE 8648, Stereoscopic Displays and Applications XXIV,
    vol. 8648, pp. 864810-1–864810-14, 2013. doi:10.1117/12.2008485 (download)
  6. Alexander Bokov, Dmitriy Vatolin, Anton Zachesov, Alexander Belous, Mikhail Erofeev,
    “Automatic detection of artifacts in converted S3D video,”
    Proc. SPIE 9011, Stereoscopic Displays and Applications XXV (March 6, 2014),
    vol. 901112, 2014. doi:10.1117/12.2054330 (download)
  7. Stanislav Dolganov, Mikhail Erofeev, Dmitriy Vatolin, Yury Gitman,
    “Detection of stuck-to-background objects in converted S3D movies,”
    2015 International Conference on 3D Imaging, IC3D 2015,
    2015. doi:10.1109/IC3D.2015.7391839 (download)
  8. Yury Gitman, Can Bal, Mikhail Erofeev, Ankit Jain, Sergey Matyunin, Kyoung-Rok Lee, Alexander Voronov, Jason Juang, Dmitriy Vatolin, Truong Nguyen,
    “Delivering Enhanced 3D Video,”
    Intel Technology Journal,
    vol. 19, pp. 162–200, 2015. (download)
  9. Dmitriy Vatolin, Alexander Bokov, Mikhail Erofeev, Vyacheslav Napadovsky,
    “Trends in S3D-Movie Quality Evaluated on 105 Films Using 10 Metrics,”
    Proceedings of Stereoscopic Displays and Applications XXVII,
    pp. SDA-439.1–SDA-439.10, 2016. doi:10.2352/ISSN.2470-1173.2016.5.SDA-439 (download)
  10. Alexander Bokov, Sergey Lavrushkin, Mikhail Erofeev, Dmitriy Vatolin, Alexey Fedorov,
    “Toward fully automatic channel-mismatch detection and discomfort prediction for S3D video,”
    2016 International Conference on 3D Imaging (IC3D),
    2016. doi:10.1109/IC3D.2016.7823462 (download)
  11. Dmitriy Vatolin, Sergey Lavrushkin,
    “Investigating and predicting the perceptibility protect of channel mismatch in stereoscopic video,”
    Moscow University Computational Mathematics and Cybernetics,
    vol. 40, pp. 185–191, 2016. doi:10.3103/s0278641916040075 (download)
  12. Anastasia Antsiferova, Dmitriy Vatolin,
    “The influence of 3D video artifacts on discomfort of 302 viewers,”
    2017 International Conference on 3D Immersion (IC3D),
    2017. doi:10.1109/IC3D.2017.8251897 (download)
  13. Dmitriy Vatolin, Alexander Bokov,
    “Sharpness Mismatch and 6 Other Stereoscopic Artifacts Measured on 10 Chinese S3D Movies,”
    Proceedings of Stereoscopic Displays and Applications XXVIII,
    pp. 137–144, 2017. doi:10.2352/ISSN.2470-1173.2017.5.SDA-340 (download)
  14. Sergey Lavrushkin, Vitaliy Lyudvichenko, Dmitriy Vatolin,
    “Local Method of Color-Difference Correction Between Stereoscopic-Video Views,”
    Proceedings of the 2018 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON),
    2018. doi:10.1109/3DTV.2018.8478453 (download)
  15. Aidar Khatiullin, Mikhail Erofeev, Dmitriy Vatolin,
    “Fast Occlusion Filling Method For Multiview Video Generation,”
    Proceedings of the 2018 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON),
    2018. doi:10.1109/3DTV.2018.8478562 (download)
  16. Sergey Lavrushkin, Dmitriy Vatolin,
    “Channel-Mismatch Detection Algorithm for Stereoscopic Video Using Convolutional Neural Network,”
    Proceedings of the 2018 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON),
    2018. doi:10.1109/3DTV.2018.8478542 (download)
  17. Alexander Ploshkin, Dmitriy Vatolin,
    “Accurate Method of Temporal Shift Estimation for 3D Video,”
    Proceedings of the 2018 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON),
    2018. doi:10.1109/3DTV.2018.8478431 (download)
  18. Sergey Lavrushkin, Konstantin Kozhemyakov, Dmitriy Vatolin,
    “Neural-Network-Based Detection Methods for Color, Sharpness, and Geometry Artifacts in Stereoscopic and VR180 Videos,”
    International Conference on 3D Immersion (IC3D),
    2020. doi:10.1109/IC3D51119.2020.9376385 (download)
  19. Kirill Malyshev, Sergey Lavrushkin, Dmitriy Vatolin,
    “Stereoscopic Dataset from A Video Game: Detecting Converged Axes and Perspective Distortions in S3D Videos,”
    International Conference on 3D Immersion (IC3D),
    2020. doi:10.1109/IC3D51119.2020.9376375 (download)
  20. Lavrushkin Sergey, Molodetskikh Ivan, Kozhemyakov Konstantin, Vatolin Dmitriy,
    “Stereoscopic quality assessment of 1,000 VR180 videos using 8 metrics,”
    Electronic Imaging, 3D Measurement and Data Processing,
    2021. doi:10.2352/issn.2470-1173.2021.2.sda-350 (download)

Reports overview

Stereo-analysis project homepage

Report 1 (S3D shooting quality analysis of 5 movies) Download
(Additional info for bloggers and press)
Pages: 246
Figures: 295
Report 2 (S3D shooting quality analysis of 5 movies) Download
(Additional info for bloggers and press)
Pages: 342
Figures: 442
Report 3 (2D-3D conversion quality analysis of 5 movies) Download
(Additional info for bloggers and press)
Pages: 305
Figures: 336
Report 4 (S3D shooting quality analysis of 5 movies) Download Pages: 301
Figures: 402
Report 5 (2D-3D conversion quality analysis of 5 movies) Download
(Additional info for bloggers and press)
Pages: 384
Figures: 404
Report 6 (Stereo Window analysis of 10 movies) Download
(Additional info for bloggers and press)
Pages: 415
Figures: 455
Report 7 (Stereo Window analysis of 10 movies) Download
(Additional info for bloggers and press)
Pages: 333
Figures: 348
Report 8 (Rotate Analysis, Temporal Shift, Channels Swap, Zoom Mismatch in 25 movies) Download
(Additional info for bloggers and press)
Pages: 366
Figures: 361
Report 9 (Temporal Shift, Stuck-to-Background Objects, 2D to S3D conversion in Captured Films) Download
(Additional info for bloggers and press)
Pages: 467
Figures: 529
Report 10 (Overall analysis of 105 movies) Download Pages: 211
Figures: 270
Report 11 (Overall analysis of 10 selected Chinese movies) Download Pages: 322
Figures: 566
Report 12 (VR180 Quality Analysis) Download Pages: 348
Figures: 362

Feedback

Do you have any feedback on previous reports? What did you find useful, or was anything lacking?

What would you like to see in future reports (analysis of certain films, different metrics, charts and visualizations, etc.)?

We are open to collaboration. If you are interested in working with us, please write your propositions:

Any other feedback that doesn't fit in previous fields:

Contacts

For questions and proposition please contact us 3dmovietest@graphics.cs.msu.ru

13 Oct 2020
See Also
Video Colorization Benchmark
Explore the best video colorization algorithms
MSU 3D-video Quality Analysis. Report 12
MSU 3D-video Quality Analysis. Report 11
Detection of stereo window violation
How to find objects that are present only in one view?
Depth continuity estimation in S3D video
How smooth is the depth transition between scenes?
Detection of 3D movie scenes shot on converged axes
Another cause of headaches when watching 3D movies.
Site structure