Main page — MSU Benchmark Collection

MSU Video Super-Resolution Benchmark: Detail Restoration — find the best upscaler

Discover the newest methods and find the most appropriate method for your tasks

Video group head: Dr. Dmitriy Vatolin
Measurements, analysis: 
Anastasia Kirillova,
Eugene Lyapustin

What’s new

Click here to see more news
  • 26.08.2021 Added 8 new algorithms
  • 15.07.2021 Improved Visualization section to be more user-friendly and added plots with the metric difference on BI and BD degradation to Leaderboard section
  • 26.04.2021 Beta-version Release

Key features of the Benchmark

  • New metrics for the detail restoration quality
    • Check methods’ ability to restore real details
  • The most complex content for restoration task: faces, text, QR-codes, car numbers, unpatterned textures, small details
    • See plots and visualizations for particular content types
  • Different degradation types to lower the resolution: bicubic interpolation (BI) and Gaussian blurring and downsampling (BD), with and without noise
    • Many methods use only one degradation type for their training datasets (e.g. bicubic interpolation) and do not work well on others. Choose the method that didn’t overfit the test dataset
  • Subjective comparison of new and popular Super-Resolution methods

Leaderboard

The table below shows a comparison of Video Super Resolution methods by subjective score and a few objective metrics. Default ranking is by subjective score. You can click on the model's name in the table to read information about the method. You can see information about all participants here.

Rank Model Subjective ERQAv1.0 LPIPS SSIM-Y** QRCRv1.0 PSNR-Y**

Charts

In this section, you can see barcharts and speed-to-performance plots. You can choose metric, motion, content, and degradation type, see tests with or without noise.
You can see information about all participants here.

Metric: Test: Content:

Metric: Test: Content:

Highlight the plot region where you want to zoom in

Visualization

In this section, you can choose a part of the frame with particular content, see a cropped piece from this, MSU VQMT PSNR* Visualization, and ERQAv1.0 Visualization for this crop. In part "QR-codes" codes, which can be detected, are surrounded by a blue rectangle. You can see information about all participants here.
*We visualize shifted PSNR metric by applying MSU VQMT PSNR Visualization to frames with optimal shift for PSNR.

Frame: Content:

Model 1: Model 2: Model 3:

Drag a red rectangle in the area, which you want to crop.

GT

D3Dnet

DBVSR

DUF-16L

Your method submission

Verify the restoration ability of your VSR algorithm and compare it with state-of-the-art solutions.
You can see information about all other participants here.

1. Download input data
Download input low-resolution videos as sequences of frames in .png format
There are 2 available options:
    a. Download 1 folder with all videos, joined in one sequence here.
    Neighboring videos are separated by 10 black frames, which will be skipped for evaluation.
    b. If you worry this strategy can decrease your performance, you can download 12 folders
    with 100 frames each here.


2. Apply your algorithm Restore high-resolution frames with your algorithm.
You can also send us the code of your method or the executable file and we will run it ourselves.


3. Send us result Send us an email to vsr-benchmark@videoprocessing.ai with the following information:
    A. Name of your method that will be specified in our benchmark
    B. Link to the cloud drive (Google Drive, OneDrive, Dropbox, etc.), containing output frames.
    Check that the number of output frames matches the number of input frames:
      1310 frames in one folder for download option (a)
      12 folders with 100 frames each for download option (b)
    C. (Optional) Execution time of your algorithm and information about used GPU
    D. (Optional) Any additional information about the method:
      1. Full name of your model
      2. The parameter set that was used
      3. Any other additional information
      4. A link to the code of your model, if it is available
      5. A link to the paper about your model
      6. Any characteristics of your model's architecture (e.g. motion compensation, propagation, fusion)

Contact Us

For questions and propositions, please contact us: vsr-benchmark@videoprocessing.ai

Cite Us

To refer to our benchmark or metric ERQA in your work, cite one of our papers:

@article{
title={Towards True Detail Restoration for Super-Resolution: A Benchmark and a Quality Metric},
author={Lyapustin, Eugene and Kirillova, Anastasia and Meshchaninov, Viacheslav and Zimin, Evgeney and Karetin, Nikolai and Vatolin, Dmitriy},
journal={arXiv preprint arXiv:2203.08923},
year={2022}
}

@conference{
author={Anastasia Kirillova. and Eugene Lyapustin. and Anastasia Antsiferova. and Dmitry Vatolin.},
title={ERQA: Edge-restoration Quality Assessment for Video Super-Resolution},
booktitle={Proceedings of the 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 4: VISAPP,},
year={2022},
pages={315-322},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010780900003124},
isbn={978-989-758-555-5},
}
26 Apr 2021
See Also
VQA Dataset
During our work we have created the database for video quality assessment with subjective scores
MSU Video Upscalers Benchmark 2022
The most extensive comparison of video super-resolution (VSR) algorithms by subjective quality
MSU Video Deblurring Benchmark 2022
Learn about the best video deblurring methods and choose the best model
Site structure
MSU Graphics & Media Lab Video Group
2019–2022