MSU HDR Video Reconstruction Benchmark 2022: HLG

The most comprehensive comparison of HDR video reconstruction methods

G&M Lab head: Dr. Dmitriy Vatolin
Measurements, analysis: 
Mikhail Voronin,
Nikolay Safonov

Key features of the Benchmark

  • The biggest comparison of HDR video restoration methods
  • A new private dataset for testing: 20 different scenes: fireworks, flowers, soccer and others
  • 10 metrics for restoration quality assessment:
    HDR-VPD-3, HDR-VQM, HDR-PSNR, HDR-SSIM, PQ-PSNR, PQ-SSIM, PQ-NIQE, PQ-VMAF, Shifted HDR-PSNR, FovVideoVDP
  • The ability to evaluate the quality yourself through the HDR video player
  • (Soon…) Subjective comparison
  • Split comparison of video restoration encoded in the two most popular gamma curves: PQ (soon...), HLG

Leaderboard

Click on the labels to sort the table
In the methodology you can read brief information about all metrics

Rank Model HDR-VQM HDR-VDP-3 HDR-PSNR HDR-SSIM PQ-NIQE PQ-PSNR PQ-SSIM PQ-VMAF Shifted HDR-PSNR FovVideoVDP
1 Kuo 0.0579 TBP 40.7642 0.9954 5.0287 34.9679 0.9862 94.0270 47.4611 9.8880
2 Maxon 0.0622 TBP 45.6478 0.9949 4.9400 35.1188 0.9846 89.9478 45.3267 9.8374
3 DeepHDR 0.1000 8.0572 33.5267 0.9907 5.0832 21.9155 0.9584 89.5382 37.0040 9.3205
4 HDRTV 0.1296 6.9813 35.9721 0.9918 5.4744 22.4636 0.9555 85.6109 26.8880 9.8788
5 KovOliv 0.1329 TBP 32.4223 0.9937 4.8269 20.1296 0.9389 91.3117 28.9170 9.8861
6 HuoPhys 0.1518 TBP 32.6570 0.9946 5.0754 18.7458 0.9383 90.0419 22.1553 9.8866
7 HDRCNN 0.1919 6.3880 33.0200 0.9663 4.9391 21.2604 0.8539 50.5081 35.0804 9.8778
8 ExpNet 0.1942 7.5910 34.0555 0.9892 5.2161 24.2994 0.9321 61.6722 40.8574 9.8887
9 Huo 0.2103 TBP 30.0900 0.9720 4.9126 23.3886 0.9527 82.4653 44.4314 9.2408
10 Akyuz 0.2521 TBP 28.1420 0.9917 4.7872 14.6789 0.8535 71.5814 23.3069 9.8799
11 SingleHDR 0.2630 8.4180 34.2872 0.9845 5.5764 25.8042 0.9606 70.6792 42.5944 9.8305

TBP – to be published

Charts

In this section you can observe the values of different metrics for each individual video and the average values In the "Video" selector, you can observe the names of the videos. For your convenience, the Dataset tab has a preview of each video and its name match.

Metric: Video:

Results

In this section, you can quickly evaluate the quality of the algorithms yourself. The video player below will show you the HDR video if your device supports this technology. We recommend opening this video via Google Chrome or Safari.

Method:

Visualization

This section presents visualizations of all algorithms.

  • The first line is a preview
  • The second line is a tonemapped crop
  • The third and fourth lines are the two exposures, which show in detail the differences in the dark and bright areas.
While shaping the exposures, we keep the visible contrast between the objects as on an HDR monitor, assuming that the expected brightness of your display is 300 nits.
Please note that we do not show GT exposures, as this information makes it possible to obtain the HDR version of the GT video.

Video:

Model 1: Model 2: Model 3:

Drag a red rectangle in the area, which you want to crop.

GT

GT

HDRTV

HDRCNN

Your method submission

Verify the restoration ability of your HDR Video Reconstruction algorithm and compare it with state-of-the-art solutions. You can see information about all other participants here.

1. Download input data
Download input SDR videos
2. Apply your algorithm Restore HDR video with your algorithm.
You can also send us the code of your method or the executable file and we will run it ourselves.
3. Send us result Send us an email to itm-benchmark@videoprocessing.ai with the following information:
    A. Name of your method that will be specified in our benchmark
    B. Link to the cloud drive (Google Drive, OneDrive, Dropbox, etc.), containing output frames.
      You can send us files in following formats:
      1) .exr, .hdr for linear gamma curve
      2) .png, .tif, if your method does the gamma curve yourself
      3) .mov, if you make a video of the frames yourself (apply the gamma curve HLG)
      Please read the evaluation section of the methodology before submitting your algorithm
    C. (Optional) Execution time of your algorithm and information about used GPU
    D. (Optional) Any additional information about the method:
      1. Full name of your model
      2. The parameter set that was used
      3. Any other additional information
      4. A link to the code of your model, if it is available
      5. A link to the paper about your model

Contacts

For questions and propositions, please contact us: itm-benchmark@videoprocessing.ai

10 May 2022
See Also
MSU Video Quality Measurement Tool: Picture types
VQMT 14.0 Online help: List of all picture types available in VQMT and their aliases
MSU HDR Video Reconstruction Benchmark Participants
The list of participants of MSU HDR Video ReconstructionBenchmark
MSU HDR Video Reconstruction Benchmark Methodology
Evaluation Methodology of MSU HDR Video Reconstruction Benchmark
MSU HDR Video Reconstruction Benchmark Dataset
MSU HDR Video Reconstruction Benchmark Dataset
MSU Super-Resolution for Video Compression Benchmark 2022
Learn about the best SR methods for compressed videos and choose the best model to use with your codec
MSU Super-Resolution for Video Compression Benchmark Participants
The list of participants of MSU Super-Resolution for Video Compression Benchmark
Site structure