MSU Video Colorization Benchmark

Explore the best video colorization algorithms!

Powered by
Powered by
G&M Lab head: Dr. Dmitriy Vatolin
Measurements, analysis: 
David Chikovani,
Sergey Lavrushkin

Our benchmark determines the best video colorization methods for propagate colors from anchor frames or auto colorize grayscale video.

Everyone is welcome to participate! Run your favorite video colorization method on our dataset and send us the result to see how well it performs. Check the Submitting section to learn the details.

Over 2000 People

have participated in the verified
pairwise subjective comparison

36 High-Resolution Test Clips

with diverse complexity of spatial
and temporal information

11 methods tested

which include video color propagation
and automatic video colorization

A lot of metrics

and their correlations
for detailed comparison

Structural Distortion Maps

for explicit artifacts detection

Speed/Quality Scatter Plots

and tables with metrics
for a comprehensive comparison

What’s New

  • March, 2024: Beta-version Release

Introduction

For a comprehensive assessment of video colorization task, we used various objective metrics and subjective comparison. Over 2000 valid participants have selected the most visually appealing colorization result in many pairwise comparison. In addition, we calculated the FPS (frames per second) to compare the speed of both algorithms and metrics.

Check the “Methodology” section to learn the details.

Scroll below for comparison charts, tables, and interactive visual comparisons of video colorization models result.

Leaderboards

Charts

Visualizations

Submitting

To add your video colorization method to the benchmark, follow these steps:

1. Download the dataset
2. Apply your video colorization method to the dataset
3. Send to david.chikovani@graphics.cs.msu.ru
the following information:
  • The full and short names of the video colorization method that we will specify in our benchmark. Also provide information about how many anchor frames were used.
  • An compressed directory with colorized frames or videos. The link must be valid within a month after receiving it. Please ensure that you permit us to download the file. By submitting this video, you agree that third parties may use it.
  • The exact commands, options, versions of used programs, etc. Each submission must contain results of exactly one model with fixed settings. Please do not fine-tune model parameters by hand for each video segment.
  • If you want us to verify your method, send us the executable, and we will run it ourselves. Then we will add the FPS of your algorithm in charts and tables. Executable's arguments must include paths to the folder with input PNG images and the folder for output PNG images. By submitting this executable, you agree that third parties may use it.

If you have any suggestions or questions, please contact us: david.chikovani@graphics.cs.msu.ru

Get Notifications About the Updates of This Benchmark

Do you want to be the first to discover the best new video colorization algorithm? We can notify you about this benchmark’s updates: simply submit your preferred email address using the form below. We promise not to send you unrelated information.

Further Reading

Check the “Methodology” section to learn how we prepare our dataset.

Check the “Metrics” section to learn more about metrics and their correlations.

Check the “Participants” section to learn which video colorization method implementations we use.


MSU Video Quality Measurement Tool

              

    The tool for performing video/image quality analyses using reference or no-reference metrics

Widest Range of Metrics & Formats

  • Modern & Classical Metrics SSIM, MS-SSIM, PSNR, VMAF and 10+ more
  • Non-reference analysis & video characteristics
    Blurring, Blocking, Noise, Scene change detection, NIQE and more

Fastest Video Quality Measurement

  • GPU support
    Up to 11.7x faster calculation of metrics with GPU
  • Real-time measure
  • Unlimited file size

  • Main MSU VQMT page on compression.ru

Crowd-sourced subjective
quality evaluation platform

  • Conduct comparison of video codecs and/or encoding parameters

What is it?

Subjectify.us is a web platform for conducting fast crowd-sourced subjective comparisons.

The service is designed for the comparison of images, video, and sound processing methods.

Main features

  • Pairwise comparison
  • Detailed report
  • Providing all of the raw data
  • Filtering out answers from cheating respondents

  • Subjectify.us
07 Nov 2023
See Also
MSU Image- and video-quality metrics analysis
Description of a project in MSU Graphics and Media Laboratory
Super-Resolution Quality Metrics Benchmark
Discover 66 Super-Resolution Quality Metrics and choose the most appropriate for your videos
Video Saliency Prediction Benchmark
Explore the best video saliency prediction (VSP) algorithms
Super-Resolution for Video Compression Benchmark
Learn about the best SR methods for compressed videos and choose the best model to use with your codec
Metrics Robustness Benchmark
Check your image or video quality metric for robustness to adversarial attacks
Video Upscalers Benchmark
The most extensive comparison of video super-resolution (VSR) algorithms by subjective quality
Site structure