Evaluation Methodology of MSU Video Frame Interpolation Benchmark 2022
Video Frame Interpolation (VFI) algorithms synthesize non-existent images between adjacent frames, with the aim of providing a smooth and consistent visual experience.
Our benchmark will rank these algorithms and determine which is the best by means of interpolation quality.
We present a new dataset for this comparison. This was done to ensure that neural network methods do not benefit from training on data that could get into our test sample.
Details about Dataset characteristics and processing you can find in Dataset tab.
PSNR – commonly used metric based on pixels’ similarity. For metric calculation, we use the implementation from MSU VQMT. A higher metric value indicates better quality.
SSIM – another commonly used metric based on structure similarity. For metric calculation, we use the implementation from MSU VQMT. A higher metric value indicates better quality.
Multiscale SSIM (MS-SSIM) is conducted over multiple scales through a process of multiple stages of sub-sampling. Implementation from Pytorch MS-SSIM. A higher metric value indicates better quality.
VMAF is a perceptual video quality assessment algorithm developed by Netflix. In our benchmark, we calculate VMAF on the Y component in YUV colorspace. For metric calculation, we use MSU VQMT.
LPIPS (Learned Perceptual Image Patch Similarity) evaluates the distance between image patches. Higher means further/more different. Lower means more similar. To calculate LPIPS we use Perceptual Similarity Metric implementation proposed in The Unreasonable Effectiveness of Deep Features as a Perceptual Metric.
The tests were performed in Google Colab. Main characteristics:
- GPU: NVIDIA K80
- reading and writing images are not taken into account
- 1920×1080 resolution
- interpolation of one frame between the same pair of adjacent frames
- result is 3rd minimum from 100 runs
For the subjective comparison we slow down outputs from algorithms in 4 times. For the participants are shown 2 seconds length videos:
- 30 fps for gaming samples
- 60 fps for others
Each one of 413 participants has seen 32 video pairs and had to choose which one of them looks more smooth (option “indistinguishable” is also available). There were 2 verification questions to protect against random answers and bots. We used these valid answers to predict the ranking using the Bradley-Terry model.
MSU Benchmark Collection
- MSU Video Upscalers Benchmark 2022
- MSU Video Deblurring Benchmark 2022
- MSU Video Frame Interpolation Benchmark 2022
- MSU HDR Video Reconstruction Benchmark 2022
- MSU Super-Resolution for Video Compression Benchmark 2022
- MSU No-Reference Video Quality Metrics Benchmark 2022
- MSU Full-Reference Video Quality Metrics Benchmark 2022
- MSU Video Alignment and Retrieval Benchmark
- MSU Mobile Video Codecs Benchmark 2021
- MSU Video Super-Resolution Benchmark
- MSU Shot Boundary Detection Benchmark 2020
- MSU Deinterlacer Benchmark
- The VideoMatting Project
- Video Completion
- Codecs Comparisons & Optimization
- MSU Datasets Collection
- Metrics Research
- Video Quality Measurement Tool 3D
- Video Filters
- Other Projects