Main page — MSU Benchmark Collection

Home Participants Evaluation methodology How to participate Contact us

MSU Deinterlacer Benchmark — selecting the best deinterlacing filter

The most comprehensive comparison of deinterlacing methods

Video group head: Dr. Dmitriy Vatolin

Measurements, analysis:

Alexey Zelentsov,
Dmitriy Konovalchuk

What’s new

26.11.2020 Beta-version Release
22.12.2020 Added VS EEDI3, VS TDeintMod, MC Deinterlacer. Tuned Kernel Deinterlacer
07.07.2021 New 2021 Dataset
01.09.2021 New Leader! DfRes 122000 G2e 3 Deinterlacer
17.09.2021 Added MFDIN, Adobe Premiere Pro Built-IN

Key features of the Benchmark

For Deinterlacing methods’ users
- Choose deinterlacing method that is the best for your speed and quality requirements
- Discover the newest deinterlacing methods’ achievements
For Researchers and Developers
- Quickly get comprehensive comparison results for your paper with our tables, visual comparison tools and performance plots
- Check the performance of your deinterlacing method on the complex cases

To submit deinterlacing method, please, follow 3 simple steps in the the Deinterlacer Submission section

We appreciate new ideas. Please, write us an e-mail to deinterlacer-benchmark@videoprocessing.ai

Leaderboard

The table below shows a comparison of deinterlacers by PSNR, SSIM metrics and by speed.

Rank	Name	PSNR	SSIM	VMAF	FPS on CPU
1.333	DfRes 122000 G2e 3	41.891	0.969	94.848	???
2.333	DfRes 121000 G2e 3	41.869	0.969	94.768	???
3.000	DfRes 61000	41.163	0.967	94.865	???
3.667	DfRes 119000	41.674	0.968	94.252	???
5.333	MFDIN	38.494	0.956	94.455	???
5.667	MSU Deinterlacer	38.556	0.963	92.141	1.300
7.667	NNEDI	37.099	0.954	93.246	1.910
8.333	Bob-Weave Deinterlacer	37.948	0.953	91.309	46.450
9.667	Vapoursynth TDeintMod	37.617	0.951	91.044	50.290
10.000	Vapoursynth EEDI3	37.061	0.954	90.122	51.900
10.000	Real-Time Deep Deinterlacer	37.031	0.953	90.964	0.270
13.000	Kernel Deinterlacer (optimal parameters)	36.449	0.947	90.757	37.910
13.667	Bob	36.668	0.951	87.242	52.830
13.667	Weston 3-Field Deinterlacer	36.788	0.947	89.625	36.750
14.667	Kernel Deinterlacer	35.712	0.939	90.250	37.850
14.667	YADIF	36.918	0.945	87.240	48.960
16.333	Motion and Area Pixel Deinterlacer	35.278	0.932	88.882	2.150
18.000	Muksun Deinterlacer	34.486	0.928	86.873	1.950
19.000	ASVZZZ Deinterlacer	34.486	0.928	86.873	1.900
20.000	PAL Interpolation	32.901	0.901	82.662	2.850
21.333	Motion Compensation Deinterlacer	29.259	0.830	64.436	1.450
21.667	Adobe Premiere Pro Built-In	30.772	0.813	57.538	6.530

Full FrameRate Leaderboard

Rank	Name	PSNR	SSIM	VMAF	FPS on CPU
1.333	DfRes 121000 G2e 3	42.184	0.969	95.254	???
2.333	DfRes 122000 G2e 3	42.157	0.969	95.242	???
3.000	DfRes 61000	41.392	0.967	95.289	???
3.333	DfRes 119000	42.024	0.968	94.812	???
5.000	MSU Deinterlacer	38.796	0.963	92.857	1.300
6.667	Bob-Weave Deinterlacer	38.226	0.953	92.211	46.450
7.667	Real-Time Deep Deinterlacer	37.266	0.954	91.860	0.270
8.000	Vapoursynth EEDI3	37.250	0.954	91.178	51.900
8.000	Vapoursynth TDeintMod	37.862	0.951	92.125	50.290
10.667	Bob	36.799	0.952	88.456	52.830
10.667	Weston 3-Field Deinterlacer	36.987	0.947	90.698	36.750
11.333	YADIF	37.152	0.945	88.454	48.960

Visualization

In this section you can see a frame, a crop from this frame, and also MSU VQMT PSNR Visualization of this crop.

Drag a red rectangle in the area which you want to crop

The frame to compare on:
Deinterlacer 1:
Deinterlacer 2:

GT

In this row you can see VQMT PSNR Visualization

DfRes 122000 G2e 3

MFDIN

Charts

Highlight the plot region where you want to zoom in

Metric:

FPS is calculated on Intel-Core i7 10700K CPU

Metric:

Sequence №: Metric:

Cross-PSNR

In this section you can see PSNR between the output of chosen deinterlacer and the others

Deinterlacer:

	PSNR

Evaluation methodology

Dataset

Our dataset is constantly updated. Now we have 28 video sequences. Each sequence contains 60 frames. Resolution of all video sequences is 1920x1080. FPS varies from 24 to 60. TFF interlacing was used to get interlaced data from GT.

Click on this text to read how exactly our current dataset was composed.

Initially, we had about 10.000 videos, downloaded from Vimeo. These videos included sports, animation, panorama, news, landscapes, parts of movies, tv shows, ads, and many other types of content. We decided to restrict the size of the dataset by a maximum of 30 videos, each containing 60 frames. Our goal was to create the most diverse dataset, respecting given restrictions.

The first goal was to cluster 10.000 videos. For this purpose, we mapped the videos into 2D space by calculating their Google Si/Ti, determined, for example, in this paper. For calculating Google Si/Ti we used FFmpeg x264 codec with options [-qp 28 -b_qfactor 1 -i_qfactor 1].
Then, we performed a simple K-Means Clustering in this 2D space to divide the videos into 30 clusters.
Then, we measured the distances between centers of clusters and all the videos from the cluster. We chose 30 videos, closest to the centers of 30 clusters. These 30 videos included some hard-cases for deinterlacers, such as running letters, scene changes, motion, etc.
Another big part of dataset composing was to choose 60-frame cuts from these 30 big videos. To do that, we performed the same process, as in steps 1-3, but now the goal was to choose 30 cuts from about 15.000 cuts. Again, we calculated Google Si/Ti for each 60-frames cut and clustered them into 30 clusters. 2 clusters contained only black and only white cuts respectfully, so we marked them as "trash" clusters.
Then, we selected top-5 cuts, closest to the center of each non-"trash" cluster.
The final step was to manually choose the best from top-5 cuts for each non-"trash" cluster.
We don't count PSNR on black frames. They were made for deinterlacers that use motion estimation (ME). Because of the black frames, ME-deinterlacer detects zero motion between two sequences and, therefore, doesn't consider motion from the previous sequence while processing the current one.

Metrics

We compare RGB frames via 3 metrics - PSNR, SSIM, and VMAF. PSNR, SSIM, and VMAF are measured over the Y-component.

For each video sequence, we take the average PSNR, SSIM, and VMAF over all frames. We decided to choose these metrics because they proved themselves to be among the best metrics to show a quality loss.

We decided to measure these metrics over the Y component because YUV is the most popular type of colorspace nowadays, but there are still a lot of versions of YUV (e.g. yuv444p, yuv420p, yuv420p12le). In these versions U and V components are different, that's why we measure only the Y component. Also, there are a lot of other color spaces, which use the Y component (e.g. YCbCr, YPbPr, UYVY, ...). Finally, we can easily compute the Y component from other color spaces, such as RGB or gray.

Here is the plot of the PSNR difference between PSNR-Y and PSNR-RGB. As we can see here, the difference is rather negligible.

Validation of deinterlacers' outputs

Another important direction of our work is to control the outputs of deinterlacers. Sometimes, it can convert colorspace, work in BFF mode instead of TFF, or maybe have bugs.

Click on this text to read how exactly we validate deinterlacers' outputs.

The main criteria are that the PSNR between GT fields and the same fields in the deinterlaced video must be equal to infinity. To control this, we make Top-Fields and Bottom-Fields plot on every second frame of GT video and deinterlaced video.

Here is the sample plot for Bob-Weave Deinterlacer. This deinterlacer passed the validation.

PSNR between Bottom-Fields of every deinterlaced frame and GT frame is equal to infinity, so we substitute infinity by zero. This means, that the bottom field exactly matches the corresponding field in the GT sequence.

Another sample plot, for MSU Deinterlacer, which had problems with colorspace.

As we can see here, Bottom-Field PSNR is about 100, but not infinity. In such cases, we make the following steps:

Convert GT data to the colorspace of the deinterlacer.
If we don't know (can't guess) which colorspace the deinterlacer uses, we form the LookUp-Table from the fields of the deinterlacer output and GT data, which must be equal to each. Then, we use this LookUp-Table to map GT to the colorspace of the deinterlacer.
In some cases, it is impossible to precisely determine LookUp-Table because the mapping is neither injective nor surjective. In such hard cases, we just choose a GT sample with the highest PSNR from the previous 2 steps.

Also, we provide MSU Video Quality Measurement Tool (VQMT) PSNR visualization of the deinterlacer output. As the last step of the validation process, we check, that the VQMT PSNR visualization is striped. It should be so because even/odds rows must exactly match the corresponding GT rows.

Here is the example of correct VQMT PSNR visualization output:

And, finally, let us take a closer look at it:

As we can see, the output is stripped, so this means that the deinterlacer passed the last validation step.

Deinterlacer Submission

To submit, you can either send us any executable file or code of your deinterlacer, or follow these 3 simple steps

Download the interlaced video here.
We have more available formats, if YUV is not suitable. Just click on this text

There are 3 available options:

Deinterlace downloaded video
The details, which may help you

Send us an email to deinterlacer-benchmark@videoprocessing.ai with the following information:

Contacts

For questions and propositions, please contact us: deinterlacer-benchmark@videoprocessing.ai

05 Nov 2020