Main page — MSU Benchmark Collection

MSU HDR Video Reconstruction Benchmark: HLG

The most comprehensive comparison of HDR video reconstruction methods

Powered by
G&M Lab head: Dr. Dmitriy Vatolin
Measurements, analysis: 
Mikhail Voronin,
Nikolay Safonov

Key features of the Benchmark

  • Comparison of 14 methods of HDR video reconstruction
  • A new private dataset for testing. 20 different scenes: fireworks, flowers, soccer and others
  • 10 metrics for restoration quality assessment
  • HDR video player for self-assessment of quality
  • Subjective comparison
  • Split comparison of video restoration with two most popular gamma curves: PQ (soon), HLG

Leaderboard

Click on the labels to sort the table.
In the methodology you can read brief information about all metrics and subjective study.
You can scroll the table to see all the results.

Rank Model Subjective HDR-VDP-3 HDR-VQM HDR-PSNR HDR-SSIM PQ-NIQE PQ-PSNR PQ-SSIM PQ-VMAF Shifted HDR-PSNR FovVideoVDP
1 GT 4.9064 0.0000 10.0000 inf 1.0000 5.6412 inf 1.0000 100.0000 inf 10.0000
2 DeepHDR 3.7604 0.1000 8.0572 33.5266 0.9907 5.0832 21.9155 0.9584 89.5382 37.0040 9.3205
3 Maxon 3.6044 0.0622 8.2153 45.6478 0.9949 4.9400 35.1188 0.9846 89.9478 45.3267 9.8374
4 HDRCNN 3.3537 0.1919 6.3880 33.0200 0.9663 4.9391 21.2604 0.8539 50.5080 35.0804 9.8778
5 SingleHDR 3.2459 0.2630 8.4180 34.2871 0.9845 5.5764 25.8042 0.9606 70.6792 42.5944 9.8305
6 HDRTV 2.7886 0.1296 6.9813 35.9721 0.9918 5.4744 22.4636 0.9555 85.6109 26.8880 9.8788
7 KovOliv 2.3654 0.1329 7.4724 32.4223 0.9937 4.8269 20.1295 0.9389 91.3118 28.9170 9.8861
8 Huo 1.7939 0.2103 8.9004 30.0900 0.9720 4.9126 23.3886 0.9527 82.4653 44.4314 9.2408
9 HDRUnet 1.3770 0.1830 6.6857 34.9894 0.9845 6.7712 24.7347 0.9388 92.1209 TBP* TBP*
10 ExpNet 1.0268 0.1942 7.5910 34.0555 0.9892 5.2161 24.2994 0.9321 61.6722 40.8574 9.8887
11 KUnet 0.6530 0.2126 6.1543 32.5082 0.9838 7.3623 24.1380 0.9236 77.6478 TBP* TBP*
12 twostageHDR 0.4650 0.1350 6.6004 31.6717 0.9864 7.7687 22.4022 0.9139 46.1830 TBP* TBP*
13 Kuo TBP* 0.0579 8.6776 40.7642 0.9954 5.0287 34.9679 0.9862 94.0270 47.4612 9.8880
14 HuoPhys TBP* 0.1518 7.0618 32.6570 0.9946 5.0754 18.7458 0.9383 90.0419 22.1553 9.8866
15 Akyuz TBP* 0.2521 6.4114 28.1420 0.9917 4.7872 14.6789 0.8535 71.5814 23.3069 9.8799

TBP* – to be published

Charts

In this section you can observe the values of different metrics for each individual video and the average values. In the "Video" selector, you can observe the names of the videos. For your convenience, the Dataset tab has a preview of each video and its full name match.

Metric: Video:

Results

In this section, you can quickly evaluate the quality of the algorithms yourself. The video player below show you the HDR video if your device supports this technology. We recommend opening this video via Google Chrome or Safari.

Method:

Visualization

This section presents a visualization of the result of each of the algorithms.

  • The first line is a preview
  • The second line is a tonemapped crop
  • The third and fourth lines are the two exposures, which show in detail the differences in the dark and bright areas
While generating the exposures, we keep the visible contrast between the objects as on an HDR monitor, assuming that the expected brightness of your display is around 300 nits.
Please note that we do not show GT exposures, as this information makes it possible to obtain the HDR version of the GT video.

Video:

Model 1: Model 2: Model 3:

Drag a red rectangle in the area, which you want to crop.

GT

GT

HDRTVNet

HDRCNN

Your method submission

Verify the restoration ability of your HDR Video Reconstruction algorithm and compare it with state-of-the-art solutions. You can see information about all other participants here.

1. Download input data
Download SDR videos
2. Apply your algorithm Convert SDR videos to HDR using your algorithm.
You can also send us the code of your method or the executable file and we will run it ourselves.
3. Send us result Send us an email to itm-benchmark@videoprocessing.ai with the following information:
    A. Name of your method that will be specified in our benchmark
    B. Link to the cloud drive (Google Drive, OneDrive, Dropbox, etc.), containing output frames.
      You can send us files in the following formats:
      1) .exr, .hdr for linear gamma curve
      2) .png, .tif, if your method does the gamma curve yourself
      3) .mov, if you make a video of the frames yourself (apply the gamma curve HLG)
      Please read the evaluation section of the methodology before submitting your algorithm
    C. (Optional) Execution time of your algorithm and information about used GPU
    D. (Optional) Any additional information about the method:
      1. Full name of your model
      2. The parameter set that was used
      3. A link to the code of your model, if it is available
      4. A link to the paper about your model, if it is available
      5. Any other additional information

You can verify the results of current participants or estimate the performance of your method on public samples of our dataset (clips "Dog" and "Rolex"). Send us an email with a request to share them with you.

Contacts

We would highly appreciate any suggestions and ideas on how to improve our benchmark. For questions and propositions, please contact us: itm-benchmark@videoprocessing.ai

Also you can subscribe to updates on our benchmark:


MSU Video Quality Measurement Tool

              

    The tool for performing video/image quality analyses using reference or no-reference metrics

Widest Range of Metrics & Formats

  • Modern & Classical Metrics SSIM, MS-SSIM, PSNR, VMAF and 10+ more
  • Non-reference analysis & video characteristics
    Blurring, Blocking, Noise, Scene change detection, NIQE and more

Fastest Video Quality Measurement

  • GPU support
    Up to 11.7x faster calculation of metrics with GPU
  • Real-time measure
  • Unlimited file size

    • Main MSU VQMT page on compression.ru

10 May 2022
See Also
PSNR and SSIM: application areas and criticism
Learn about limits and applicability of the most popular metrics
Learning-Based Image Compression Benchmark
The First extensive comparison of Learned Image Compression algorithms
Super-Resolution for Video Compression Benchmark
Learn about the best SR methods for compressed videos and choose the best model to use with your codec
Video Colorization Benchmark
Explore the best video colorization algorithms
Defenses for Image Quality Metrics Benchmark
Explore defenses from adv attacks
Super-Resolution Quality Metrics Benchmark
Discover 66 Super-Resolution Quality Metrics and choose the most appropriate for your videos
Site structure
MSU Graphics & Media Lab Video Group
2019–2022