MSU Benchmark Collection

News

• 06.04.2022[MSU Video Super Resolution Benchmark: Detail Restoration] Added 8 new algorithms and LPIPS metric
• 16.03.2022[MSU Video Super Resolution Benchmark: Detail Restoration] Preprint of our paper "Towards True Detail Restoration for Super-Resolution: A Benchmark and a Quality Metric" was released on arXiv
• 15.11.2021[MSU Video Super Resolution Benchmark: Detail Restoration] Our paper "ERQA: Edge-restoration Quality Assessment for Video Super-Resolution" was accepted to VISAPP 2022
• 09.11.2021 [MSU Video Upscalers Benchmark 2021] Alpha-version Release
• 26.10.2021 [MSU Super-Resolution for Video Compression Benchmark 2021] Updated the Methodology
• 22.10.2021 [MSU Video Alignment and Retrieval Benchmark] Public beta-version Release
• 12.10.2021 [MSU Super-Resolution for Video Compression Benchmark 2021] Published October Report. Added 2 new videos to the dataset. Updated Charts section and Visualizations
• 06.10.2021 [MSU Video Alignment and Retrieval Benchmark] Alpha-version Release
• 01.10.2021 [MSU Mobile Video Codecs Benchmark 2021] Beta-version Release
• 14.09.2021 [MSU Super-Resolution for Video Compression Benchmark 2021] Public beta-version Release
• 31.08.2021 [MSU Super-Resolution for Video Compression Benchmark 2021] Alpha-version Release
• 26.04.2021 [MSU Video Super Resolution Benchmark: Detail Restoration] Beta-version Release
• 05.05.2021 [MSU Shot Boundary Detection Benchmark 2020] Main Release
</ul>

Released benchmarks list

1. MSU Video Upscalers Benchmark 2021
2. MSU Video Alignment and Retrieval Benchmark
3. MSU Super-Resolution for Video Compression Benchmark 2021
4. MSU Mobile Video Codecs Benchmark
5. MSU Video Super Resolution Benchmark: Detail Restoration
6. MSU Shot Boundary Detection Benchmark 2020
7. MSU Deinterlacer Benchmark
8. The VideoMatting Project
9. Video Completion
10. (soon) MSU Video Deblurring Benchmark
11. (soon) MSU Video Frame Interpolation Benchmark
12. (soon) MSU Full Reference Video Quality Metric Benchmark
13. (soon) MSU No Reference Video Quality Metric Benchmark
14. (soon) MSU Super Precision Benchmark

Super-Resolution Benchmarks

MSU Super-Resolution for Video Compression Benchmark 2021




With the emergence of new video resolution standards, more efficient video encoding and decoding techniques are required. Our benchmark can help determine the best SR models to work with each of the different codec standards. This information will help make video coding with downsampling more effective.

Key features

• Subjective comparison with more than 1900 valid participants
• Different objective metrics ranked by their correlation with the subjective assessment
• 75 SR+codec pairs

MSU Video Super Resolution Benchmark: Detail Restoration




Super-Resolution is the process of calculating high-resolution samples from their low-resolution counterparts. Working with images we can utilize natural preferences and make a high-resolution image, which is only in a way similar to the real one. Our benchmark is aimed to find the best algorithms for the restoration of real details with Video Super-Resolution.

Key features

• Subjective comparison with more than 1900 valid participants
• ERQAv1.0 metric
• 32 Methods

MSU Video Upscalers Benchmark




Super-Resolution is the process of calculating high-resolution samples from their low-resolution counterparts. We want to create the most comprehensive comparison of video super-resolution (VSR) algorithms by subjective quality

Key features

• Subjective comparison with more than 4300 valid participants
• Check how upscalers behave in the most practical upscale use cases

Other benchmarks

MSU Mobile Video Codecs Benchmark




Measurement of the speed and power efficiency of different codecs on different mobile platforms allows for deeper understanding of their suitability for different devices, and allows manufacturers to fine-tune their codec integration

Key features

• Speed and power efficiency measurement
• Video playback time increase by up to 22 hours
• 147 Android models, 6 compression standards.

MSU Shot Boundary Detection Benchmark 2020




One of the basic steps in video processing is video scene splitting. For example, scene cutting is a necessary step in video annotation and indexing, keyframe searching, and automatic video format changing. Our benchmark is aimed at measuring the performance of video scene splitting algorithms

Key features

• Extensive and diverse datasets
• Beautiful and easy-interpreting visualizations

MSU Deinterlacer Benchmark




Deinterlacing is the process of converting interlaced video into a non-interlaced or progressive form. Interlaced video signals are commonly found in analog television, digital television, some DVD titles, and a smaller number of Blu-ray discs. Our benchmark is aimed at measuring the performance of video deinterlacing algorithms

Key features

• Сhoose deinterlacing method that is the best for your speed and quality requirements
• Discover the newest deinterlacing methods’ achievements

MSU Video Alignment and Retrieval Benchmark




Often, broadcasted video sequences can have some freeze frames. Because of this, the process of comparing the initial sequence and the result one is very obstructed. Video alignment aims at finding point correspondences between two video sequences to overcome this problem. Our benchmark is aimed at measuring the performance of video alignment algorithms

Key features

• 4 Methods
• 3 tracks varying on distortions type
• 560 test pairs in each track with a total duration of ~2 million frames

The Video­Mat­ting pro­ject




The Video­Mat­ting pro­ject is the first pub­lic ob­jec­tive bench­mark for video-mat­ting meth­ods. We be­lieve our work will help rank ex­ist­ing meth­ods and aid de­vel­op­ers of new meth­ods in im­prov­ing their re­sults.

Key features

• Green screen dataset
• Stop motion dataset

Video completion




The Video­Com­ple­tion pro­ject in­tro­duces the first bench­mark for video-com­ple­tion meth­ods. We pre­sent re­sults for dif­fer­ent meth­ods on a range of di­verse test se­quences which are avail­able for view­ing on a player equipped with a mov­able zoom re­gion. We be­lieve that our work can help rank ex­ist­ing meth­ods and as­sist de­vel­op­ers of new gen­eral-pur­pose video-com­ple­tion meth­ods.

Key features

• 7 video se­quences
• Different objective metrics

Planned benchmarks

MSU Video Deblurring Benchmark




Blur often obscures the process of extracting details. Our bechmark aims at measuring the performance of details restoration by modern deblurring algorithms

Key features

• Large subjective comparison
• Dataset with real blur
• 2 Tracks: motion deblurring and defocus deblurring

MSU Video Frame Interpolation Benchmark




A low frame rate causes aliasing, yields abrupt motion artifacts, and degrades the video quality. To solve this problem a lot of video frame interpolation algorithms have been created so far. Our benchmark will rank these algorithms and determine which is the best by means of interpolation quality.

Key features

• Large subjective comparison
• The most comprehensive comparison of frame interpolation algorithms

Feedback

What benchmark are you most interested in?
What benchmark would you like to see in the future?
Email (optional)
Name (optional)

About our benchmarks

The development of benchmarks is important for many reasons:
• They motivate developers to create cool new methods in this scientific field
• There are few high-quality and constantly updated benchmarks in some scientific fields. Our mission is to fix it
• PSNR and SSIM are not suitable for video comparison anymore. Our goal is to prove it to everyone
• Benchmark creation is a first step in developing new metrics, better than PSNR and SSIM (and even VMAF!). It is much easier to develop a metric, if you have the benchmark