MSU No-Reference Video Quality Metrics Benchmark 2022
Discover the newest metrics and find the most appropriate method for your tasks
Diverse dataset
- 40 different video codecs of 10 compression standards
- 2500+ compressed streams
- 780.000+ subjective scores
- 10.000+ viewers
- User-generated content
VQA and IQA metrics
- 20+ metrics without variations
- The biggest leaderboard of neural networks-based video quality metrics
- Calculations over U and V planes
- Metrics with different weighted
average for planes
Various charts
- Bar chart with the overall metrics perfomance
- Comparison on different compression standards with 95% confidence intervals
- Speed-Quality chart
Note
This page is a part of MSU Video Quality Benchmark, which you can find here.
Results
The chart below shows the correlation of metrics with subjective scores on our dataset. You can choose the type of correlation and compression standard of codecs used for compression. We recommend that you focus on Spearman’s rank correlation coefficient.
Correlation type: Compression Standard:
The results of the comparison on different compression standards and different bitrates ranges, as well as full-reference and no-reference metrics detailed analysis, are presented on the leaderboard page.
Methodology and dataset
To see all steps of metrics evaluation and the description of our dataset visit the methodology page.
How to submit your method
Find out the strong and weak sides of your method and compare it to the best commercial and free methods.
We kindly invite you to participate in our benchmark. To do this follow the steps below:
Send us an email to vqa@videoprocessing.ai
with the following information:
ref — path to reference video (for full-reference metrics)
dist — path to distorted video
output — path to output of your algorithm
t — threshold, if it's required in your algorithm
|
You can verify the results of current participants or estimate the perfomance of your method on public samples
of our dataset. Just send us an email with a request to share them with you. |
Our policy:
- We won't publish the results of your method without your permission.
- We share only public samples of our dataset as it is private.
Information about all other participants you can find in the participants page.
Cite us
@inproceedings{
antsiferova2022video,
title={Video compression dataset and benchmark of learning-based video-quality metrics},
author={Anastasia Antsiferova and Sergey Lavrushkin and Maksim Smirnov and Aleksandr Gushchin and Dmitriy S. Vatolin and Dmitriy Kulikov},
booktitle={Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
year={2022},
url={https://openreview.net/forum?id=My5AI9aM49R}
}
|
You can find the full text of our paper through the link.
Contacts
We would highly appreciate any suggestions and ideas on how to improve our benchmark. Please contact us via email: vqa@videoprocessing.ai.
Also you can subscribe to updates on our benchmark:
-
MSU Benchmark Collection
- MSU Video Upscalers Benchmark 2022
- MSU Video Deblurring Benchmark 2022
- MSU Video Frame Interpolation Benchmark 2022
- MSU HDR Video Reconstruction Benchmark 2022
- MSU Super-Resolution for Video Compression Benchmark 2022
- MSU No-Reference Video Quality Metrics Benchmark 2022
- MSU Full-Reference Video Quality Metrics Benchmark 2022
- MSU Video Alignment and Retrieval Benchmark
- MSU Mobile Video Codecs Benchmark 2021
- MSU Video Super-Resolution Benchmark
- MSU Shot Boundary Detection Benchmark 2020
- MSU Deinterlacer Benchmark
- The VideoMatting Project
- Video Completion
- Codecs Comparisons & Optimization
- VQMT
- MSU Datasets Collection
- Metrics Research
- Video Quality Measurement Tool 3D
- Video Filters
- Other Projects