Metrics correlations of the Video Colorization Benchmark

Metrics charts

Metrics

PSNR

PSNR is a commonly used metric for reconstruction quality for images and video. In our benchmark, we calculate PSNR on the A, B components in LAB colorspace.

SSIM

SSIM is a metric based on structural similarity. In our benchmark, we calculate SSIM on the A, B components in LAB colorspace.

LPIPS

LPIPS (Learned Perceptual Image Patch Similarity) evaluates the distance between image patches. Higher means further/more different. Lower means more similar.

Colorfulness

Colorfulness is a no reference metric that is proposed to evaluate the colorfulness of an image. It uses statistics calculated on A, B components in LAB colorspace.

Warp Error

Warp Error is a metric for evaluating the temporal stability of a video. By warping using optical flow of one frame to another, we can compare their differences. In our benchmark, we calculate WarpError on the A, B components in LAB colorspace.

CDC

CDC (the Color Distribution Consistency index), which measures the Jensen–Shannon (JS) divergence of the color distribution between consecutive frames. Unlike the commonly-used warping error, CDC is specifically designed for the video colorization task, and can better reflect the consistency of color.

ID

Fréchet inception distance (FID) is a metric for quantifying the realism and diversity of images generated by generative adversarial networks (GANs). FID has a frequent use in articles on colorization, but we want to point out that this metric makes sense for unpaired datasets. For paired datasets it is enough to calculate just Inception Distance.

Metrics' runtime

We measured runtime of metrics, for cpu-compatible metrics (PSNR, SSIM, Color, CDC, WE) we run on AMD EPYC 7532 32-Core Processor @ 1.50 GHz, for gpu-based metrics (LPIPS, ID) we run on NVIDIA RTX A6000.

07 Nov 2023
See Also
MSU Image- and video-quality metrics analysis
Description of a project in MSU Graphics and Media Laboratory
Super-Resolution Quality Metrics Benchmark
Discover 66 Super-Resolution Quality Metrics and choose the most appropriate for your videos
Video Saliency Prediction Benchmark
Explore the best video saliency prediction (VSP) algorithms
Super-Resolution for Video Compression Benchmark
Learn about the best SR methods for compressed videos and choose the best model to use with your codec
Metrics Robustness Benchmark
Check your image or video quality metric for robustness to adversarial attacks
Video Upscalers Benchmark
The most extensive comparison of video super-resolution (VSR) algorithms by subjective quality
Site structure