Main page — MSU Benchmark Collection

MSU HDR Video Reconstruction Benchmark 2022: HLG

The most comprehensive comparison of HDR video reconstruction methods

G&M Lab head: Dr. Dmitriy Vatolin
Measurements, analysis: 
Mikhail Voronin,
Nikolay Safonov

Key features of the Benchmark

  • Comparison of 14 methods of HDR video reconstruction
  • A new private dataset for testing. 20 different scenes: fireworks, flowers, soccer and others
  • 10 metrics for restoration quality assessment
  • HDR video player for self-assessment of quality
  • (Soon) Subjective comparison
  • Split comparison of video restoration with two most popular gamma curves: PQ (soon), HLG

Leaderboard

Click on the labels to sort the table.
In the methodology you can read brief information about all metrics.
You can scroll the table to see all the results.

Rank Model HDR-VDP-3 HDR-VQM HDR-PSNR HDR-SSIM PQ-NIQE PQ-PSNR PQ-SSIM PQ-VMAF Shifted HDR-PSNR FovVideoVDP
1 Kuo 8.6776 0.0579 40.7642 0.9954 5.0287 34.9679 0.9862 94.0270 47.4611 9.8880
2 Maxon 8.2153 0.0622 45.6478 0.9949 4.9400 35.1188 0.9846 89.9478 45.3267 9.8374
3 DeepHDR 8.0572 0.1000 33.5267 0.9907 5.0832 21.9155 0.9584 89.5382 37.0040 9.3205
4 HDRTV 6.9813 0.1296 35.9721 0.9918 5.4744 22.4636 0.9555 85.6109 26.8880 9.8788
5 KovOliv 7.4724 0.1329 32.4223 0.9937 4.8269 20.1296 0.9389 91.3117 28.9170 9.8861
6 twostageHDR 6.6004 0.1350 31.6717 0.9864 7.7687 22.4022 0.9139 46.1830 TBP TBP
7 HuoPhys 7.0618 0.1518 32.6570 0.9946 5.0754 18.7458 0.9383 90.0419 22.1553 9.8866
8 HDRUNet 6.6857 0.1830 34.9894 0.9845 6.7712 24.7347 0.9388 92.1209 TBP TBP
9 HDRCNN 6.3880 0.1919 33.0200 0.9663 4.9391 21.2604 0.8539 50.5081 35.0804 9.8778
10 ExpNet 7.5910 0.1942 34.0555 0.9892 5.2161 24.2994 0.9321 61.6722 40.8574 9.8887
11 Huo 8.9004 0.2103 30.0900 0.9720 4.9126 23.3886 0.9527 82.4653 44.4314 9.2408
12 KUnet 6.1543 0.2126 32.5082 0.9838 7.3623 24.1380 0.9236 77.6478 TBP TBP
13 Akyuz 6.4114 0.2521 28.1420 0.9917 4.7872 14.6789 0.8535 71.5814 23.3069 9.8799
14 SingleHDR 8.4180 0.2630 34.2872 0.9845 5.5764 25.8042 0.9606 70.6792 42.5944 9.8305

TBP* – to be published

Charts

In this section you can observe the values of different metrics for each individual video and the average values. In the "Video" selector, you can observe the names of the videos. For your convenience, the Dataset tab has a preview of each video and its full name match.

Metric: Video:

Results

In this section, you can quickly evaluate the quality of the algorithms yourself. The video player below show you the HDR video if your device supports this technology. We recommend opening this video via Google Chrome or Safari.

Method:

Visualization

This section presents a visualization of the result of each of the algorithms.

  • The first line is a preview
  • The second line is a tonemapped crop
  • The third and fourth lines are the two exposures, which show in detail the differences in the dark and bright areas
While generating the exposures, we keep the visible contrast between the objects as on an HDR monitor, assuming that the expected brightness of your display is around 300 nits.
Please note that we do not show GT exposures, as this information makes it possible to obtain the HDR version of the GT video.

Video:

Model 1: Model 2: Model 3:

Drag a red rectangle in the area, which you want to crop.

GT

GT

HDRTVNet

HDRCNN

Your method submission

Verify the restoration ability of your HDR Video Reconstruction algorithm and compare it with state-of-the-art solutions. You can see information about all other participants here.

1. Download input data
Download SDR videos
2. Apply your algorithm Convert SDR videos to HDR using your algorithm.
You can also send us the code of your method or the executable file and we will run it ourselves.
3. Send us result Send us an email to itm-benchmark@videoprocessing.ai with the following information:
    A. Name of your method that will be specified in our benchmark
    B. Link to the cloud drive (Google Drive, OneDrive, Dropbox, etc.), containing output frames.
      You can send us files in the following formats:
      1) .exr, .hdr for linear gamma curve
      2) .png, .tif, if your method does the gamma curve yourself
      3) .mov, if you make a video of the frames yourself (apply the gamma curve HLG)
      Please read the evaluation section of the methodology before submitting your algorithm
    C. (Optional) Execution time of your algorithm and information about used GPU
    D. (Optional) Any additional information about the method:
      1. Full name of your model
      2. The parameter set that was used
      3. A link to the code of your model, if it is available
      4. A link to the paper about your model, if it is available
      5. Any other additional information

Contacts

We would highly appreciate any suggestions and ideas on how to improve our benchmark. For questions and propositions, please contact us: itm-benchmark@videoprocessing.ai

Also you can subscribe to updates on our benchmark:

10 May 2022
See Also
MSU CVQAD – Compressed VQA Dataset
During our work we have created the database for video quality assessment with subjective scores
MSU Video Upscalers Benchmark 2022
The most extensive comparison of video super-resolution (VSR) algorithms by subjective quality
MSU Video Deblurring Benchmark 2022
Learn about the best video deblurring methods and choose the best model
Site structure
MSU Graphics & Media Lab Video Group
2019–2022