Metric: PSNR SSIM VMAF MOS

Metric: PSNR SSIM VMAF MOS

Metric: PSNR SSIM VMAF

Metric: PSNR SSIM VMAF MOS

Sequence №: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 Metric: PSNR SSIM VMAF MOS

Sequence №: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 Metric: PSNR SSIM VMAF

Deinterlacer: Adobe Premiere Pro ASVZZZ Deinterlacer Bob Bob-Weave Deinterlacer Real-Time Deep Deinterlacer Motion and Area Pixel Deinterlacer DfRes 122000 G2e 3 DfRes DfRes SA DUF EDVR EDVR_woTSA Vapoursynth EEDI3 FLAD Kernel Deinterlacer Kernel Deinterlacer (optimal parameters) Motion Compensation Deinterlacer MFDIN MFDIN L MSU Deinterlacer Muksun Deinterlacer NNEDI PAL Interpolation SonyVegas Blend Field SonyVegas Interpolate Field ST-Deint SwinDI TDAN Vapoursynth TDeintMod Weston 3-Field Deinterlacer YADIF

Dataset

Our dataset is constantly updated. Now we have 28 video sequences. Each sequence contains 60 frames. Resolution of all video sequences is 1920x1080. FPS varies from 24 to 60. TFF interlacing was used to get interlaced data from GT.

Metrics

We compare RGB frames via 3 metrics - PSNR, SSIM, and VMAF. PSNR, SSIM, and VMAF are measured over the Y-component.

For each video sequence, we take the average PSNR, SSIM, and VMAF over all frames. We decided to choose these metrics because they proved themselves to be among the best metrics to show a quality loss.

We decided to measure these metrics over the Y component because YUV is the most popular type of colorspace nowadays, but there are still a lot of versions of YUV (e.g. yuv444p, yuv420p, yuv420p12le). In these versions U and V components are different, that's why we measure only the Y component. Also, there are a lot of other color spaces, which use the Y component (e.g. YCbCr, YPbPr, UYVY, ...). Finally, we can easily compute the Y component from other color spaces, such as RGB or gray.

Here is the plot of the PSNR difference between PSNR-Y and PSNR-RGB. As we can see here, the difference is rather negligible.

Subjective comparison

Also, we provide MOS (Mean Opinion Score) for the best 16 deinterlacers (excluding different versions of the same deinterlacer).

To get MOS, we host a subjective comparison using www.subjectify.us. We get the same crops (of areas with the worst PSNR) from each test video, and then assessors compare output of each deinterlacer against all others. We get 10 comparison results for each crop-crop pair, and then we use Bradley-Terry model to get MOS scores

Validation of deinterlacers' outputs

Another important direction of our work is to control the outputs of deinterlacers. Sometimes, it can convert colorspace, work in BFF mode instead of TFF, or maybe have bugs.

Click on this text to read how exactly we validate deinterlacers' outputs.

The main criteria are that the PSNR between GT fields and the same fields in the deinterlaced video must be equal to infinity. To control this, we make Top-Fields and Bottom-Fields plot on every second frame of GT video and deinterlaced video.

Here is the sample plot for Bob-Weave Deinterlacer. This deinterlacer passed the validation.

PSNR between Bottom-Fields of every deinterlaced frame and GT frame is equal to infinity, so we substitute infinity by zero. This means, that the bottom field exactly matches the corresponding field in the GT sequence.

Another sample plot, for MSU Deinterlacer, which had problems with colorspace.

As we can see here, Bottom-Field PSNR is about 100, but not infinity. In such cases, we make the following steps:

1. Convert GT data to the colorspace of the deinterlacer.
2. If we don't know (can't guess) which colorspace the deinterlacer uses, we form the LookUp-Table from the fields of the deinterlacer output and GT data, which must be equal to each. Then, we use this LookUp-Table to map GT to the colorspace of the deinterlacer.
3. In some cases, it is impossible to precisely determine LookUp-Table because the mapping is neither injective nor surjective. In such hard cases, we just choose a GT sample with the highest PSNR from the previous 2 steps.

Also, we provide MSU Video Quality Measurement Tool (VQMT) PSNR visualization of the deinterlacer output. As the last step of the validation process, we check, that the VQMT PSNR visualization is striped. It should be so because even/odds rows must exactly match the corresponding GT rows.

Here is the example of correct VQMT PSNR visualization output:

And, finally, let us take a closer look at it:

As we can see, the output is stripped, so this means that the deinterlacer passed the last validation step.

To submit, you can either send us any executable file or code of your deinterlacer, or follow these 3 simple steps

1. Download the interlaced video here.
   We have more available formats, if YUV is not suitable. Just click on this text

   
   There are 3 available options:

   a. Download frames of the video sequence in .png format here

   b. Download .yuv video file generated from frames via

   ffmpeg -i %04d.png -c:v rawvideo -pix_fmt yuv444p sequences.yuv

   here

   c. Download lossless encoded .mkv video generated from frames via

   ffmpeg -i %04d.png -c:v libx264 -preset ultrafast -crf 0 -pix_fmt yuv444p lossless.mkv

   here


2. Deinterlace downloaded video
   The details, which may help you
   TFF interlacing was used to get interlaced sequence from GT
   The video consists of 28 videos, each separated by 5 black frames. Black frames are ignored while measuring
   
   


3. Send us an email to deinterlacer-benchmark@videoprocessing.ai with the following information:
   A. Name of the deinterlacing method that will be specified in our benchmark
   
   
   B. Link to the cloud drive (Google Drive, OneDrive, Dropbox, etc.), containing deinterlaced video.
   
   
   C. (Optional) Any additional information about the method
   Click here to see what may be included in additional information
   Technical information about deinterlaced video (e.g. colorspace, file-type, codec)
   The name of the theoretical method used
   Full name of the deinterlacing method or product
   The version that was used
   The parameter set that was used
   Any other additional information
   A link to the code of your deinterlacing method, if it is open-source
   A link to the paper about your deinterlacing method
   A link to the documentation of your deinterlacing method. For example, this is suitable for deinterlacing methods that are implemented as a part of a video processing framework.
   A link to the page, where users can purchase or download your product (for example, VirtualDub Plugin)
   
   
   D. (Optional) If you would like us to tune the parameters of your deinterlacing method, you should give us an ability to launch it. You can do it by sending us a code or an executable file, providing us a free test version of your product, or any other possible way, that is convenient for you