Important Links

• Our GitHub Repository

• Our Benchmark on PapersWithCode

What’s new

• 16.06.2023 The alpha version of Benchmark page is released
• 05.12.2023 Benchmark Release!
• 05.05.2024 Benchmark Release on GitHub and PapersWithCode, added Q-Align (VQA, IQA and IAA), added LIQE and LIQE-MIX

Introduction

Our benchmark provides a ranking of Super-Resolution quality evaluation metrics.

This rating is based on the correlation between the values of metrics with subjective scores. All of these scores were obtained from pairwise comparisons of different distorted videos with a single reference video (ground-truth, GT).

Scroll down for comparison charts, tables, and interactive visual comparisons of metric performance.

Leaderboard Table

In this section, you can see the leaderboard of the metrics. The metrics' values were calculated on the videos from our four datasets (details are in the "Methodology" section).
You can see information about all participants here.

Leaderboard Chart

This is the same leaderboard, but presented as a barchart with 95% confidence intervals.

Runtime Comparison

In this section, you can compare the performance of each metric with its correlation with subjective scores. For clarity, the chart shows the Pareto front.

How to participate

Find out the strong and weak sides of your metric and compare it to the best commercial and free metrics. We kindly invite you to participate in our benchmark. To do this follow the steps below:

Download the dataset from Google Drive or here (for a description of the dataset format, see the following link) Run your metric on it Send us an email to artem.borisov@graphics.cs.msu.ru with the following information: A. Results of the metric on this dataset in the format shown here B. The name of your metric that will be specified in our benchmark C. Your metric and a script to run it Launch script must have the following (or similar) options --ref — path to reference video/image (for full-reference metrics) --dist — path to distorted video/image --output — path to output of your metric D. (Optional) Any additional information about the metric: 1. The parameter set that you want us to use 2. A link to the paper about your model 3. Any characteristics of your model's architecture

If you have any suggestions or questions, please contact us.

Contacts

We would highly appreciate any suggestions and ideas on how to improve our benchmark. Please contact us via e-mail: artem.borisov@graphics.cs.msu.ru.
Also you can subscribe to updates on our benchmark: