Main page — MSU Benchmark Collection
Home Participants Evaluation methodology How to participate Contact us

MSU Deinterlacer Benchmark — selecting the best deinterlacing filter

The most comprehensive comparison of deinterlacing methods

Video group head: Dr. Dmitriy Vatolin
Measurements, analysis: 
Alexey Zelentsov,
Dmitriy Konovalchuk

What’s new

Key features of the Benchmark

To submit deinterlacing method, please, follow 3 simple steps in the the Deinterlacer Submission section

We appreciate new ideas. Please, write us an e-mail to deinterlacer-benchmark@videoprocessing.ai

Leaderboard

The table below shows a comparison of deinterlacers by PSNR, SSIM metrics and by speed.

Rank Name MOS PSNR SSIM VMAF FPS on CPU
1.000 MFDIN L No MOS 42.547 0.977 97.429 1.580
2.000 DfRes SA No MOS 42.177 0.969 95.177 0.020
3.333 DfRes 122000 G2e 3 0.884 41.891 0.969 94.848 ???
4.333 DfRes 61000 0.869 41.163 0.967 94.865 ???
5.667 DfRes V No MOS 39.446 0.968 93.585 0.640
6.667 EDVR_woTSA No MOS 39.686 0.961 93.327 0.580
7.000 DfRes No MOS 39.247 0.968 93.212 0.420
8.667 EDVR No MOS 39.337 0.959 92.805 0.560
9.000 MFDIN 1.026 38.494 0.956 94.455 1.580
9.000 MSU Deinterlacer 0.767 38.556 0.963 92.141 1.300
10.667 DUF No MOS 38.512 0.957 91.595 0.740
12.000 NNEDI 0.509 37.099 0.954 93.246 1.910
13.333 Bob-Weave Deinterlacer 0.378 37.948 0.953 91.309 46.450
15.333 Real-Time Deep Deinterlacer 0.595 37.031 0.953 90.964 0.270
15.667 Vapoursynth EEDI3 0.436 37.061 0.954 90.122 51.900
15.667 Vapoursynth TDeintMod 0.276 37.617 0.951 91.044 50.290
16.333 TDAN No MOS 37.625 0.952 89.876 0.680
19.000 SonyVegas Interpolate Field 0.184 36.649 0.951 90.151 3.310
19.333 Kernel Deinterlacer (optimal parameters) 0.105 36.449 0.947 90.757 37.910
20.000 Bob 0.148 36.668 0.951 87.242 52.830
20.333 Weston 3-Field Deinterlacer 0.245 36.788 0.947 89.625 36.750
21.000 Kernel Deinterlacer 0.093 35.712 0.939 90.250 37.850
21.333 YADIF 0.678 36.918 0.945 87.240 48.960
23.333 Motion and Area Pixel Deinterlacer No MOS 35.278 0.932 88.882 2.150
25.000 ASVZZZ Deinterlacer No MOS 34.486 0.928 86.873 1.900
26.000 Muksun Deinterlacer 0.386 34.486 0.928 86.873 1.950
27.000 PAL Interpolation No MOS 32.901 0.901 82.662 2.850
28.667 Motion Compensation Deinterlacer No MOS 29.259 0.830 64.436 1.450
29.000 Adobe Premiere Pro Built-In No MOS 30.772 0.813 57.538 6.530
29.333 SonyVegas Blend Field No MOS 28.344 0.856 49.308 3.510

Full FrameRate Leaderboard

Rank Name MOS PSNR SSIM VMAF FPS on CPU
1.000 DfRes SA No MOS 42.487 0.969 95.617 0.020
2.333 DfRes 122000 G2e 3 0.884 42.157 0.969 95.242 ???
3.333 DfRes 61000 0.869 41.392 0.967 95.289 ???
4.000 DfRes V No MOS 39.697 0.968 94.096 0.640
5.000 DfRes No MOS 39.510 0.968 93.809 0.420
6.000 EDVR_woTSA No MOS 39.653 0.959 93.492 0.580
7.000 EDVR No MOS 39.581 0.958 93.278 0.560
7.333 MSU Deinterlacer 0.767 38.796 0.963 92.857 1.300
9.000 DUF No MOS 38.728 0.957 92.365 0.740
11.000 TDAN No MOS 38.259 0.955 91.561 0.680
11.333 Bob-Weave Deinterlacer 0.378 38.226 0.953 92.211 46.450
12.333 Real-Time Deep Deinterlacer 0.595 37.266 0.954 91.860 0.270
12.667 Vapoursynth TDeintMod 0.276 37.862 0.951 92.125 50.290
13.000 Vapoursynth EEDI3 0.436 37.250 0.954 91.178 51.900
15.667 Weston 3-Field Deinterlacer 0.245 36.987 0.947 90.698 36.750
15.667 Bob 0.148 36.799 0.952 88.456 52.830
16.333 YADIF 0.678 37.152 0.945 88.454 48.960

Visualization

In this section you can see a frame, a crop from this frame, and also MSU VQMT PSNR Visualization of this crop.

Drag a red rectangle in the area which you want to crop

The frame to compare on:
Deinterlacer 1:
Deinterlacer 2:

GT

In this row you can see VQMT PSNR Visualization

DfRes 122000 G2e 3

MFDIN

Charts

Highlight the plot region where you want to zoom in

Metric:

Metric:

FPS is calculated on Intel-Core i7 10700K CPU

Metric:

Metric:

Sequence №: Metric:

Sequence №: Metric:

Cross-PSNR

In this section you can see PSNR between the output of chosen deinterlacer and the others

Deinterlacer:

PSNR

Evaluation methodology

Dataset

Our dataset is constantly updated. Now we have 28 video sequences. Each sequence contains 60 frames. Resolution of all video sequences is 1920x1080. FPS varies from 24 to 60. TFF interlacing was used to get interlaced data from GT.

Click on this text to read how exactly our current dataset was composed.
Initially, we had about 10.000 videos, downloaded from Vimeo. These videos included sports, animation, panorama, news, landscapes, parts of movies, tv shows, ads, and many other types of content. We decided to restrict the size of the dataset by a maximum of 30 videos, each containing 60 frames. Our goal was to create the most diverse dataset, respecting given restrictions.
  1. The first goal was to cluster 10.000 videos. For this purpose, we mapped the videos into 2D space by calculating their Google Si/Ti, determined, for example, in this paper. For calculating Google Si/Ti we used FFmpeg x264 codec with options [-qp 28 -b_qfactor 1 -i_qfactor 1].
  2. Then, we performed a simple K-Means Clustering in this 2D space to divide the videos into 30 clusters.
  3. Then, we measured the distances between centers of clusters and all the videos from the cluster. We chose 30 videos, closest to the centers of 30 clusters. These 30 videos included some hard-cases for deinterlacers, such as running letters, scene changes, motion, etc.
  4. Another big part of dataset composing was to choose 60-frame cuts from these 30 big videos. To do that, we performed the same process, as in steps 1-3, but now the goal was to choose 30 cuts from about 15.000 cuts. Again, we calculated Google Si/Ti for each 60-frames cut and clustered them into 30 clusters. 2 clusters contained only black and only white cuts respectfully, so we marked them as "trash" clusters.
  5. Then, we selected top-5 cuts, closest to the center of each non-"trash" cluster.
  6. The final step was to manually choose the best from top-5 cuts for each non-"trash" cluster.
  7. We don't count PSNR on black frames. They were made for deinterlacers that use motion estimation (ME). Because of the black frames, ME-deinterlacer detects zero motion between two sequences and, therefore, doesn't consider motion from the previous sequence while processing the current one.

Metrics

We compare RGB frames via 3 metrics - PSNR, SSIM, and VMAF. PSNR, SSIM, and VMAF are measured over the Y-component.

For each video sequence, we take the average PSNR, SSIM, and VMAF over all frames. We decided to choose these metrics because they proved themselves to be among the best metrics to show a quality loss.

We decided to measure these metrics over the Y component because YUV is the most popular type of colorspace nowadays, but there are still a lot of versions of YUV (e.g. yuv444p, yuv420p, yuv420p12le). In these versions U and V components are different, that's why we measure only the Y component. Also, there are a lot of other color spaces, which use the Y component (e.g. YCbCr, YPbPr, UYVY, ...). Finally, we can easily compute the Y component from other color spaces, such as RGB or gray.

Here is the plot of the PSNR difference between PSNR-Y and PSNR-RGB. As we can see here, the difference is rather negligible.

PSNR-Y_PSNR-RGB_plot

Subjective comparison

Also, we provide MOS (Mean Opinion Score) for the best 16 deinterlacers (excluding different versions of the same deinterlacer).

To get MOS, we host a subjective comparison using www.subjectify.us. We get the same crops (of areas with the worst PSNR) from each test video, and then assessors compare output of each deinterlacer against all others. We get 10 comparison results for each crop-crop pair, and then we use Bradley-Terry model to get MOS scores

Validation of deinterlacers' outputs

Another important direction of our work is to control the outputs of deinterlacers. Sometimes, it can convert colorspace, work in BFF mode instead of TFF, or maybe have bugs.

Click on this text to read how exactly we validate deinterlacers' outputs.

The main criteria are that the PSNR between GT fields and the same fields in the deinterlaced video must be equal to infinity. To control this, we make Top-Fields and Bottom-Fields plot on every second frame of GT video and deinterlaced video.

Here is the sample plot for Bob-Weave Deinterlacer. This deinterlacer passed the validation.

BWDIF validation

PSNR between Bottom-Fields of every deinterlaced frame and GT frame is equal to infinity, so we substitute infinity by zero. This means, that the bottom field exactly matches the corresponding field in the GT sequence.

Another sample plot, for MSU Deinterlacer, which had problems with colorspace.

MSU Deinterlacer validation

As we can see here, Bottom-Field PSNR is about 100, but not infinity. In such cases, we make the following steps:

  1. Convert GT data to the colorspace of the deinterlacer.
  2. If we don't know (can't guess) which colorspace the deinterlacer uses, we form the LookUp-Table from the fields of the deinterlacer output and GT data, which must be equal to each. Then, we use this LookUp-Table to map GT to the colorspace of the deinterlacer.
  3. In some cases, it is impossible to precisely determine LookUp-Table because the mapping is neither injective nor surjective. In such hard cases, we just choose a GT sample with the highest PSNR from the previous 2 steps.

Also, we provide MSU Video Quality Measurement Tool (VQMT) PSNR visualization of the deinterlacer output. As the last step of the validation process, we check, that the VQMT PSNR visualization is striped. It should be so because even/odds rows must exactly match the corresponding GT rows.

Here is the example of correct VQMT PSNR visualization output:

MSU Deinterlacer validation

And, finally, let us take a closer look at it:

MSU Deinterlacer validation

As we can see, the output is stripped, so this means that the deinterlacer passed the last validation step.

Deinterlacer Submission

To submit, you can either send us any executable file or code of your deinterlacer, or follow these 3 simple steps

  1. Download the interlaced video here.
    We have more available formats, if YUV is not suitable. Just click on this text
    There are 3 available options:
      a. Download frames of the video sequence in .png format here
      b. Download .yuv video file generated from frames via

      ffmpeg -i %04d.png -c:v rawvideo -pix_fmt yuv444p sequences.yuv

      here
      c. Download lossless encoded .mkv video generated from frames via

      ffmpeg -i %04d.png -c:v libx264 -preset ultrafast -crf 0 -pix_fmt yuv444p lossless.mkv

      here

  2. Deinterlace downloaded video
    The details, which may help you
      TFF interlacing was used to get interlaced sequence from GT
      The video consists of 28 videos, each separated by 5 black frames. Black frames are ignored while measuring


  3. Send us an email to deinterlacer-benchmark@videoprocessing.ai with the following information:
      A. Name of the deinterlacing method that will be specified in our benchmark

      B. Link to the cloud drive (Google Drive, OneDrive, Dropbox, etc.), containing deinterlaced video.

      C. (Optional) Any additional information about the method
      Click here to see what may be included in additional information
        Technical information about deinterlaced video (e.g. colorspace, file-type, codec)
        The name of the theoretical method used
        Full name of the deinterlacing method or product
        The version that was used
        The parameter set that was used
        Any other additional information
        A link to the code of your deinterlacing method, if it is open-source
        A link to the paper about your deinterlacing method
        A link to the documentation of your deinterlacing method. For example, this is suitable for deinterlacing methods that are implemented as a part of a video processing framework.
        A link to the page, where users can purchase or download your product (for example, VirtualDub Plugin)

      D. (Optional) If you would like us to tune the parameters of your deinterlacing method, you should give us an ability to launch it. You can do it by sending us a code or an executable file, providing us a free test version of your product, or any other possible way, that is convenient for you

Contacts

For questions and propositions, please contact us: deinterlacer-benchmark@videoprocessing.ai

05 Nov 2020
See Also
PSNR and SSIM: application areas and critics
Learn about limits and applicability of the most popular metrics
MSU Video Upscalers Benchmark 2021
The most comprehensive comparison of video super resolution (VSR) algorithms by subjective quality
MSU Video Upscalers Benchmark Participants
The list of the participants of the MSU Video Upscalers Benchmark
MSU Video Upscalers Benchmark Methodology
The methodology of the MSU Video Upscalers Benchmark
MSU Super-Resolution for Video Compression Benchmark
Learn about the best SR methods for compressed videos and choose the best model to use with your codec
MSU Super-Resolution for Video Compression Benchmark Participants
The list of participants of MSU Super-Resolution for Video Compression Benchmark
Site structure
MSU Graphics & Media Lab Video Group
2019–2020