MSU Benchmark Collection


News

Released benchmarks list

  1. Learning-Based Image Compression Benchmark
  2. Super-Resolution Quality Metrics Benchmark
  3. MSU Video Saliency Prediction Benchmark
  4. MSU Video Upscalers Benchmark: Quality Enhancement
  5. MSU SR+Codecs Benchmark
  6. MSU Video Frame Interpolation Benchmark
  7. MSU VSR Benchmark: Detail Restoration
  8. MSU Metric Robustness Benchmark
  9. MSU FR Video Quality Metric Benchmark
  10. MSU NR Video Quality Metric Benchmark
  11. MSU Mobile Video Codecs Benchmark
  12. MSU Deinterlacer Benchmark
  13. MSU HDR Video Reconstruction Benchmark
  14. MSU BASED: Video Deblurring Benchmark
  15. MSU Video Alignment and Retrieval Benchmark
  16. MSU Shot Boundary Detection Benchmark
  17. MSU The VideoMatting Project
  18. MSU Video Completion

Super-Resolution Benchmarks

Super-Resolution Quality Metrics Benchmark


Our benchmark features the most extensive comparison of existing image and video quality metrics relative for Super-Resolution task and determines the best of them. Everyone is welcome to participate!

Key features

  • 66 Super-Resolution Metrics for different tasks
  • Comparison on 1187 videos
  • Regular leaderboard updates

Video Upscalers Benchmark: Quality Enhancement


Our benchmark determines the best upscaling methods for increasing video resolution and improving visual quality using our compact yet comprehensive dataset and features the most extensive comparison of video super-resolution (VSR) algorithms by subjective quality. Everyone is welcome to participate! Run your favorite super-resolution method on our compact test video and send us the result to see how well it performs.

Key features

  • Over 3700 people have participated in the verified pairwise subjective comparison
  • 30 test clips with both camera-shot and 2D-animated content
  • 41 upscalers tested with both 4× and 2× scaling on video with complex distortion

SR+Codecs Benchmark


With the emergence of new video resolution standards, more efficient video encoding and decoding techniques are required. Our benchmark can help determine the best SR models to work with each of the different codec standards. This information will help make video coding with downsampling more effective.

Key features

  • Subjective comparison with more than 1900 valid participants
  • Different objective metrics ranked by their correlation with the subjective assessment
  • 75 SR+codec pairs

VSR Benchmark: Detail Restoration


Super-Resolution is the process of calculating high-resolution samples from their low-resolution counterparts. Working with images we can utilize natural preferences and make a high-resolution image, which is only in a way similar to the real one. Our benchmark is aimed to find the best algorithms for the restoration of real details with Video Super-Resolution.

Key features

  • Subjective comparison with more than 1900 valid participants
  • ERQAv1.0 metric
  • 32 Methods

Other benchmarks

Learning-Based Image Compression Benchmark


Our benchmark is a comparison of the best learned and conventional image compression methods. Last years learning based image compression methods show expressive results in Rate-Distortion performance. Our benchmark is the first large comparison of learned image compression methods. Everyone welcome to participate!

Key features

  • Over 750 test images
  • HD, Full HD and 4K resolutions
  • 19 codecs tested
  • 13 IQA metrics
  • Regular leaderboard updates

Video Saliency Prediction Benchmark


Our benchmark is a comparison of the best video saliency prediction methods for recognition of the most important areas of the video. It is based on a high-resolution multitype dataset collected from observers using eye-tracker. The comparison methodology includes Domain adaptation with brightness change and Center Prior blending for generalization of blurring of initial fixations.

Key features

  • Adapting models to our dataset for a more objective comparison and improving the portability of predictions
  • 41 high-resolution test clips of 3 types
  • Reliable data collection using 500 Hz eye-tracker
    for 50 observers
  • Model results open visualizations for comparison
  • 28 models tested in 15 various works

Metrics Robustness Benchmark


Image and video quality assessment plays a key role in optimal media compression. Neural-network-based methods show higher performance than traditional methods, however they also became more vulnerable to adversarial attacks, that increase the metric without improving visual quality. We collected more than 15 IQA/VQA methods, adapted adversarial attacks on classifiers, and compared methods' robustness.

Key features

  • More than 15 no-reference image/video-quality metrics
  • 9 adversarial attacks
  • 6 training and testing datasets
  • Automatic cloud-based pipeline

Video BASED: Video Deblurring Benchmark


Deblurring is the process of removing blurring artifacts from images. Video deblurring recovers a sharp sequence from a blurred one. Current SOTA aproaches use deep learning algorithms for this task. Our benchmark ranks these algorithms and determines which is the best by means of restoration quality.

Key features

  • A new private real motion blur dataset for testing: 23 different scenes
  • Comparison of 9 methods of video deblurring: VRT, NAFNet and more
  • 5 metrics for restoration quality assessment

FR Video Quality Metrics Benchmark


Video-quality measurement is a critical task in video processing. We present a new benchmark for video-quality metrics that evaluates video compression. It is based on a new dataset consisting of 2500+ streams encoded using different standards, including AVC, HEVC, AV1, and VVC. The list of evaluated metrics includes recent ones based on machine learning and neural networks.

Key features

  • Diverse dataset with 40+ codecs and 2500+ compressed streams
  • Subjective comparison with more then 10000 viewers and 780000+ subjective scores
  • 20+ metrics with different variations

NR Video Quality Metrics Benchmark


Video-quality measurement is a critical task in video processing. We present a new benchmark for video-quality metrics that evaluates video compression. It is based on a new dataset consisting of 2500+ streams encoded using different standards, including AVC, HEVC, AV1, and VVC. The list of evaluated metrics includes recent ones based on machine learning and neural networks.

Key features

  • Diverse dataset with 40+ codecs and 2500+ compressed streams
  • Subjective comparison with more then 10000 viewers and 780000+ subjective scores
  • 20+ metrics with different variations

HDR Video Reconstruction Benchmark


HDR restoration is the process of creating an HDR video from its SDR version by restoring brightness. Our benchmark evaluates the quality of HDR video recovery from SDR using various algorithms. This benchmark will help you find the method with the most natural restoration of HDR video.

Key features

  • Comparison of 20 methods of HDR video reconstruction
  • A new private dataset for testing. 20 different scenes: fireworks, flowers, soccer and others
  • 10 metrics for restoration quality assessment

Mobile Video Codecs Benchmark


Measurement of the speed and power efficiency of different codecs on different mobile platforms allows for deeper understanding of their suitability for different devices, and allows manufacturers to fine-tune their codec integration

Key features

  • Speed and power efficiency measurement
  • Video playback time increase by up to 22 hours
  • 147 Android models, 6 compression standards.

Shot Boundary Detection Benchmark


One of the basic steps in video processing is video scene splitting. For example, scene cutting is a necessary step in video annotation and indexing, keyframe searching, and automatic video format changing. Our benchmark is aimed at measuring the performance of video scene splitting algorithms

Key features

  • Extensive and diverse datasets
  • Beautiful and easy-interpreting visualizations

Deinterlacer Benchmark


Deinterlacing is the process of converting interlaced video into a non-interlaced or progressive form. Interlaced video signals are commonly found in analog television, digital television, some DVD titles, and a smaller number of Blu-ray discs. Our benchmark is aimed at measuring the performance of video deinterlacing algorithms

Key features

  • Сhoose deinterlacing method that is the best for your speed and quality requirements
  • Discover the newest deinterlacing methods’ achievements

Video Alignment and Retrieval Benchmark


Often, broadcasted video sequences can have some freeze frames. Because of this, the process of comparing the initial sequence and the result one is very obstructed. Video alignment aims at finding point correspondences between two video sequences to overcome this problem. Our benchmark is aimed at measuring the performance of video alignment algorithms

Key features

  • 4 Methods
  • 3 tracks varying on distortions type
  • 560 test pairs in each track with a total duration of ~2 million frames

The Video­Mat­ting pro­ject


The Video­Mat­ting pro­ject is the first pub­lic ob­jec­tive bench­mark for video-mat­ting meth­ods. We be­lieve our work will help rank ex­ist­ing meth­ods and aid de­vel­op­ers of new meth­ods in im­prov­ing their re­sults.

Key features

  • Green screen dataset
  • Stop motion dataset

Video completion


The Video­Com­ple­tion pro­ject in­tro­duces the first bench­mark for video-com­ple­tion meth­ods. We pre­sent re­sults for dif­fer­ent meth­ods on a range of di­verse test se­quences which are avail­able for view­ing on a player equipped with a mov­able zoom re­gion. We be­lieve that our work can help rank ex­ist­ing meth­ods and as­sist de­vel­op­ers of new gen­eral-pur­pose video-com­ple­tion meth­ods.

Key features

  • 7 video se­quences
  • Different objective metrics

Video Frame Interpolation Benchmark


A low frame rate causes aliasing, yields abrupt motion artifacts, and degrades the video quality. To solve this problem a lot of video frame interpolation algorithms have been created so far. Our benchmark will rank these algorithms and determine which is the best by means of interpolation quality.

Key features

  • Large subjective comparison
  • The most comprehensive comparison of frame interpolation algorithms

Feedback

About our benchmarks

The development of benchmarks is important for many reasons:

Super-Resolution Quality Metrics Benchmark
Discover 66 Super-Resolution Quality Metrics and choose the most appropriate for your videos
Learning-Based Image Compression Benchmark
The First extensive comparison of Learned Image Compression algorithms
Video Saliency Prediction Benchmark
Explore the best video saliency prediction (VSP) algorithms
Super-Resolution for Video Compression Benchmark
Learn about the best SR methods for compressed videos and choose the best model to use with your codec
Metrics Robustness Benchmark
Check your image or video quality metric for robustness to adversarial attacks
Video Upscalers Benchmark
The most extensive comparison of video super-resolution (VSR) algorithms by subjective quality
Video Deblurring Benchmark
Learn about the best video deblurring methods and choose the best model
Video Frame Interpolation Benchmark
Discover the best algorithm to make high-quality and smooth slow motion videos
HDR Video Reconstruction Benchmark
The most comprehensive comparison of HDR video reconstruction methods
No-Reference Video Quality Metrics Benchmark
Explore newest No-Reference Video Quality Metrics and find the most appropriate for you.
Full-Reference Video Quality Metrics Benchmark
Explore newest Full-Reference Video Quality Metrics and find the most appropriate for you.
Video Alignment and Retrieval Benchmark
Explore the best algorithms in different video alignment tasks
Mobile Video Codecs Benchmark
Discover Android devices with the longest video playback time and find the most power-efficient video-decoder on your Android device and
Video Super-Resolution Benchmark
Discover the newest VSR methods and find the most appropriate method for your tasks
Shot Boundary Detection Benchmark
Discover the best Shot Boundary Detection method for your case
Site structure