Overview

Brief Description

MSU Video Quality Measurement Tool (MSU VQMT) is professional software that is used to perform deep comparative objective analysis of video quality. The main functionality of this software is to calculate objective quality metrics for digital multimedia content (video or image) using reference (when comparing one or several processed/compressed/distorted video sequences to original one) or non-reference (when analyzing content and getting mark of its quality) types of analysis.

MSU VQMT is developed by COMPRESSION.RU Team with participants MSU Graphics & Media Lab. Video Group (G&M Lab.). Project leader: dr. Dmitry Vatolin.

To use MSU VQMT with Python refer to MSU VQMT Python Interface.

See also: MSU VQMT online help on https://videoprocessing.ai.

Main support email: video-measure@compression.ru.

Main application areas

Codec developers Modern metrics are close to human perception. Control quality loss with VQMT.

Television industry. Be sure in picture quality.

Studios. Control quality in automatic or manual mode on stage of production

Medical equipment developers. Analysis quality and parameters of medical devices signal.

Video providers. Reduce storage and transmission cost.

Research groups & Universities. Use VQMT to control arbitrary parameters of video and images.

Individuals. Writers, bloggers, journalists in video segment.

AV system manufacturers. For camera, video compressors, transmitters developers.

We use MSU VQMT in own projects, for example:

• Moscow State University, CS MSU Graphics & Media Lab. Annual Codecs Comparison https://www.compression.ru/video/codec_comparison/index_en.html.

• MSU Benchmarks. https://videoprocessing.ai/benchmarks/

• See also x264 Codec Capabilities analysis from YUVsoft Corp.
  http://yuvsoft.com/pdf/x264_parameters_comparison.pdf.

Editions & API

FREE license is an edition for individuals. It supplies only GUI edition for Windows and has limitations.

DEMO license is an edition for testing PRO/Premium features. This version is useless - computed values are incorrect. However it can be used to test VQMT features and performance.

Trial license is an edition for companies considering purchasing VQMT. This edition can be requested on VQMT website. The standard trial period is 2 weeks.

PRO license is an edition for companies and individuals, who wants to use all abilities of MSU VQMT GUI and CLI interfaces on Windows or Linux. It helps them to carry out massive comparisons of their technologies using batch processing of big numbers of files and flexible options for measurements without technical limitations. One PRO verison license grants access to one Windows and one Linux platform simultaneously. We have very attractive volume discounts.

PREMIUM license is an edition for companies, who actively uses MSU VQMT. It doesn’t limit installations, supports automatical activation, using in clouds, on virtual machines and more.

SDK license is a special expensive license for companies that want to incorporate VQMT into their workflow.

System Requirements

Supported Windows systems (GUI and CLI):

• Microsoft Windows 7 x64 SP 1 or later, with installed updates: KB2999226, KB976932, KB2533623.
• Microsoft Windows 8 x64, 10 x64 or later.

Supported Linux systems (command-line interface only):

• Debian family (Ubuntu 16.04 or later)
• Red Hat family (CentOS 7 or later)
• Support for other standalone Linux systems – by request.

Hardware requirements:

• 2 GHZ x86-64 Processor
• Processor with AVX support required on Linux
• 4 GB of RAM
• 200 Mb free disc space

We recommend:

• 4+ core processor with AVX support
• 8 GM RAM
• Graphical adapter with at least 1 GB video RAM.

We recommend Premium version for usage in clouds, containers, virtual machines and with other virtualization technologies. The PRO activation is guaranteed to work only on standalone workstations, laptops.

Main Features

Three types of User Interface:

1. Command-Line Interface (Windows & Linux, Professional only);
2. Graphic User Interface (Windows only, Free & Professional);
3. Python Interface. Can be installed via PyPI package.

Various input video formats:

• Video files (*.avi; *.avs; *.dat; *.divx; *.f4v; *.flv; *.h265; *.m2ts; *.m2v; *.m4v; *.mkv; *.mov; *.mp4; *.mpg; *.mts; *.mxf; *.ogm; *.ogv; *.qt; *.ts; *.vob; *.wmv; *.265; *.3g2; *.3gp; *.3gpp and others);
• Raw files with 1-16 bit integer color depth or 32-bit floats, different colorspaces and subsamplings;
• Image files and image sequences: *.bmp, *.bw, *.cut, *.dds, *.exr, *.g3, *.gif (only static images), *.hdp, *.hdr, *.ico, *.iff, *.j2c, *.j2k, *.jif, *.jng, *.jp2, *.jpe, *.jpeg, *.jpg, *.jxr, *.lbm, *.mng, *.pbm, *.pcd, *.pct, *.pcx, *.pgm, *.pic, *.pict, *.png, *.ppm, *.psd, *.ras, *.rgb, *.rgba, *.sgi, *.targa, *.tga, *.tif, *.tiff, *.wap, *.wbm, *.wbmp, *.wdp, *.webp, *.xbm, *.xpm;
• Full-HD, 4K, 8K and other resolutions support;
• TIFF images with 16 and 32 bit (floating point) color depth;
• AviSynth scripts – it can be very useful when using VOB, WMV and other files as input for MSU VQMT;
• Video files via AviSynth scripts auto generation;

Base metrics are included:

1. PSNR;
2. HDR PSNR;
3. MSE;
4. MSAD;
5. Delta;
6. Delta ICtCp;
7. SSIM (superfast, fast, precise, OpenCL, CUDA);
8. HDR SSIM;
9. MSSSIM (superfast, fast, precise, OpenCL, CUDA);
10. HDR MS-SSIM;
11. 3SSIM (CPU, OpenCL, CUDA);
12. stSSIM (temporary excluded);
13. VQM;
14. HDR VQM;
15. NIQE (no-reference);
16. VMAF;
17. SI (no-reference)
18. TI (no-reference)

Additional objective quality metrics from MSU:

• MSU Brightness Flicking Metric (with source code);
• MSU Brightness Independent PSNR (with source code);
• MSU Drop Frame Metric (with source code);
• MSU Noise Estimation Metric (with source code);
• MSU Scene Change Detector (with source code);
• MSU Blocking (no-reference);
• MSU Blurring (no-reference);

In command-line mode metrics can be calculated simultaneously. Also metrics supports:

• Sets of color components;
• ROI;
• Saving results to CSV of JSON files
• Visualization;
• Different averaging modes;
• Plug-ins interface with SDK that gives user possibility to create their own metrics;
• Possibility to compare several files in one comparison – typical situation when comparing original uncompressed file with several compressed files with different codecs or presets;
• Possibility to save “bad” frames for every comparison with flexible options – it can be very useful when comparing two (or more) codecs and user wants to see frames where these codecs have maximum difference, the lowest/highest metric value, etc.

Changelog

14.1 (June 2022). Python wrapper

14.0 BETA (Jan 2022). SSIM, MS-SSIM Superfast metrics

13.1 (May 2021).

13.0 BETA (Feb 2021). Colorspaces, HDR metrics, AV1

12.1 (Feb 2021). VMAF on OpenCL, Non-binary masks

12.0 BETA r12434 (Jan 2020).

12.0 BETA (Nov 2019). Online metrics, acceleration

11.2 (Nov 2019).

11.1 (May 2019). Geometry correction

11.0 (Feb 2019). 1400+ raw formats, wider HDR

10.2 (Feb 2019).

10.1 (Apr 2018). New metrics: VMAF, NIQE

10.0 BETA (Apr 2017). New main window, Linux support

9.1 (Apr 2017).

9.0 BETA (Dec 2016). New result window

8.1 (Nov 2016).

8.0 BETA (Oct 2016). Preview window

7.1 (Oct 2016).

7.0 BETA (Jun 2016). OpenCL support, Metrics reorganized

6.2 (Jun 2016). JSON support

6.1 BETA (Apr 2016).

6.0 BETA (Mar 2016). Acceleration

5.2 (Apr 2016).

5.1 (Mar 2016). Ranges and offsets support

5.0 BETA (Sep 2015). Open wizard, Command line refactor

4.4 (Sep 2015). GPU support improved

4.3 BETA (May 2015).

4.2 BETA (Apr 2015).

4.1 BETA (Mar 2015). Native reading video files via FFmpeg

3.2 (Feb 2015). AviSynth support improved

3.1 (Nov 2012).

3.0 (Jun 2011). GPU support, 64-bit version

2.7.3 (Oct 2010).

2.7.2 (Aug 2010).

2.7.1 (Jun 2010).

2.7 (May 2010). MSSSIM, 3SSIM metrics added

2.6 (Jan 2010). Windows Vista & Windows 7 support

2.5 (Nov 2009).

2.01 beta (Apr 2009). Masking added

2.0 BETA (Mar 2009). HDTV support, Deep RAW files

1.0 (May 2006).

Command-line Help

Alphabet index

Help, information

Getting help on command-line options, information about VQMT features, lists of metrics, supported formats and devices.

Input

Specifying video input files, streams, pipes, or image sequences. Setting options, colorspaces and formats for opening these files.

Metrics

Setting up metrics to apply to the input files. Setting color components for analysis and configuring metrics. Selection of devices.

Output

Setting up the output to JSON, CSV, stdout, log and other files that will contain the results of the analysis.

Visualization

Setting the output of per-pixel metric values. Visualization will generate an image for each metric and each processed frame.

Masking

Setting region of interest (ROI) to indicate areas which the metric should consider with bigger or lesser weight. Can be an input video/image or set by parameters.

Bad frames

Requesting output of frames of input video/image with extreme metric values. VQMT will save the requested number of images containing input frames.

Preprocessing

Correction of images before applying metric. Setting of the scaling to bring the images to the same size or reduce the resolution.

Indexing

Preliminary analysis of input streams to improve the quality of seeking and accurate information about the progress of the analysis.

Sample interpretation

Additional setting of working color standards, transformations of input samples, screen luminance for HDR images.

Performance

Adjusting CPU usage and memory consumption, skipping frames. Requesting output detailes information of VQMT performance after the analysis.

Misc.

Performing VQMT activation or reactivation. Interface selection. Specifying a project file or JSON configuration file. Configuring output profile.

JSON configuration description

The configuration is a JSON file that fully specifies VQMT computation settings. It specifies the set of files, the set of metrics, and calculation settings. You can use the configuration file instead of a classical CLI configuration (for any computational purpose, excluding getting help, specification and activation). Using the configuration file is possible in command via the specifying -config option. Also, it is possible to read the configuration JSON from STDIN using -config-stdin. The configuration is available only in the CLI interface in PRO/Premium edition.

• version - This configuration file version (optional)

  Syntax: [integer,integer]

  First integer for major version, second one for minor. Supported versions:

  • 12.1
  • 13.0
  • 13.1
  • 14.0
  • 14.1

  Default: [14,1]

• files - List of all input files. The first will be treated as the original one for reference metrics

  Syntax: [A, ...]. A possible values: One of:

  • “string”
  • [“string”, {object}]

  The string is always file name. Possible keys for the object:

  • mode - (string) - explicit mode for reading a file. See also List of open modes. Default: “auto”
  • index - (string) - index mode
  • index_file - (string) - index file. For some index modes will be skipped
  • props - (object) - additional properties, specific for mode and file (see List of open modes)
  • range - (array) - specify array of one element - [N] to skip N frames. Specify array of two elements [N, M] to select frames from N to M inclusive
  • stdin - (bool) - specify true, if the file should be read from stdin
  • offset_type - (string) - if range specified, the way for seeking target frame

• metrics - List of metrics to perform

  Syntax: [A, ...]. A possible values: {object}

  • metric - Metric to perform

    Syntax:

    One of:

    • “string”
    • {“name”:”string”, “device”:”string”}
    • {“name”:”string”, “variation”:”string”}

    Run VQMT with -list json metrics arguments to see the list of all available metrics. Each metric can be uniquely identified by it’s name and variation

  • color - The color component to perform metric. You can specify only one component per element.

    Syntax: “string”

    Run VQMT with -list json metrics arguments to see the list of all available metrics with supported color components

  • config - Metric configuration (optional)

    Syntax: {object}

    Run VQMT with -list json metrics arguments to see the list of all available metrics with available configuration settings

    Default: {}

  • vis-gamma - Gamma correction value for visualization (optional)

    Syntax: null or A, where A possible values: float

    Default: null

  • vis-colormap - Colormap for visualization (optional)

    Syntax: null or A, where A possible values: “string”

    Default: null

• output - Settings for writing JSON ans CSV files

  Syntax: {object}

  • enabled - Turn on/off saving output

    Syntax:

    One of:

    • true
    • false

    Default: true

  • output_directory - Output directory for saving CSV and JSON (supports VQMT environment variables)

    Syntax: “string”

    Default: “?(InputOriginalDir)”

  • name_generation_type - Scheme for default naming of output files (CSV, JSON)

    Syntax:

    One of:

    • “prefix”
    • “postfix”
    • “custom”

    Default: “postfix”

  • custom_name - Custom name for output files (supports VQMT environment variables). Only considered if the scheme is CUSTOM

    Syntax: “string”

    Default: “”

  • csv_separator - Separator for starting new cell in the output CSV file

    Syntax:

    One of:

    • ”;”
    • ”,”

    Default: “,”

  • fp_separator - Separator for starting fractional part in floating point numbers for the output CSV file

    Syntax:

    One of:

    • ”.”
    • ”,”

    Default: “.”

  • save_csv - Turning on/off sabing CSV

    Syntax:

    One of:

    • true
    • false

    Default: false

  • save_json - Turning on/off saving JSON

    Syntax:

    One of:

    • true
    • false

    Default: false

  • json_legacy - Switching to JSON format of older VQMT releases. Currently supported: 11

    Syntax: integer

    Default: 14

  • json_file - Explicit name of the JSON file (supports VQMT environment variables). If set and not empty, has a priority over other specifications. It will override custom_name if set

    Syntax: “string”

    Default: “”

  • csv_file - Explicit name of the CSV file (supports VQMT environment variables). If set and not empty, has a priority over other specifications. It will override custom_name if set

    Syntax: “string”

    Default: “”

  • json_stdout - Enabling/disabling saving JSON to STDOUT instead of a file. Do not use with json_file

    Syntax:

    One of:

    • true
    • false

    Default: false

• vis - Settings for writing visualization

  Syntax: {object}

  • enabled - Turn on/off saving visualization for all metrics

    Syntax:

    One of:

    • true
    • false

    Default: false

  • output_directory - Output directory for saving visualization (supports VQMT environment variables)

    Syntax: “string”

    Default: “?(InputOriginalDir)”

  • name_generation_type - Scheme for default naming of visualization files

    Syntax:

    One of:

    • “prefix”
    • “postfix”
    • “custom”

    Default: “postfix”

  • compressor - Compressor name for visualization files. See also the list of all compressors

    Syntax: “string”

    Default: “h264”

  • preset - Preset for setting compressor mode

    Syntax: “string”

    Default: “mq”

  • caption_position - Place for printing caption over the visualization (if some data are printed)

    Syntax:

    One of:

    • “left-top”
    • “left-bottom”
    • “right-top”
    • “right-bottom”

    Default: “left-top”

  • write_value - Turning on/off printing metric value on each visualization frame

    Syntax:

    One of:

    • true
    • false

    Default: true

  • write_frame - Turning on/off printing frame number on each visualization frame

    Syntax:

    One of:

    • true
    • false

    Default: true

  • gamma - The default gamma correction value for visualization. Can be overwritten by each metric individually by specifying the metric setting

    Syntax: float

    Default: 1.0000000000000000

  • colormap - The default color table for visualization. Can be overwritten by each metric individually by specifying the metric setting

    Syntax: “string”

    Default: “”

• conversion - Settings for interpreting samples of input files

  Syntax: {object}

  • standard_name - Color standard that will be used to find exact colorspace for files, where metadata not sufficient to specify exact colorspace

    Syntax: “string”

    Default: “BT.709”

  • full_range - Is quantization range PC (full range, 0..255 in case of 8-bit samples). Quantization range will be used to find exact colorspace representation for files, where metadata not sufficient to specify coded data quantization range

    Syntax:

    One of:

    • true
    • false

    Default: false

  • display_lum - Display luminance for some HDR colorspaces and metrics. This will be treated as the luminance of a maximum white pixel of an input file. This value will be also used in some metrics as maximum. Value is integer in the range 1..10000

    Syntax: float

    Default: 10000.000000000000

  • sample_conv - The way an integer sample will be converted to continuous

    Syntax:

    One of:

    • “shift” - 0xFF means 0.996 (255/256)
    • “repeat” - 0xFF means 1.0

    Default: “repeat”

  • legacy_mode - Emulation of an old VQMT mode. It affects some metrics and conversion algorithm. Do not use with other settings of this section

    Syntax: integer

    Default: 0

  • disable_UV_upscale - Disabling upscaling of chroma components (U, V for YUV pixtures types) to Y size for subsampled formats (YUV420, YUV422, etc.)

    Syntax:

    One of:

    • true
    • false

    Default: false

• bad_frames - Settings for bad frames

  Syntax: {object}

  • enabled - Turn on/off saving visualization for all metrics

    Syntax:

    One of:

    • true
    • false

    Default: false

  • dir - Directory for saving bad frames (suports VQMT environment variables)

    Syntax: “string”

    Default: “?(InputOriginalDir)”

  • count - Total (maximum) number of bad frames to be saved

    Syntax: integer

    Default: 10

  • distance - The minimum distance that bad frames can be (frames)

    Syntax: integer

    Default: 0

  • type - The criterion for the selection of bad frames

    Syntax:

    One of:

    • “origproc”
    • “procproc”
    • “firstbetter”
    • “secondbetter”

    Default: “origproc”

  • format - Image format for saving bad frames

    Syntax:

    One of:

    • “BMP”
    • “TIFF”

    Default: “TIFF”

  • high_threshold_enabled - Is high threshold enabled

    Syntax:

    One of:

    • true
    • false

    Default: false

  • low_threshold_enabled - Is low threshold enabled

    Syntax:

    One of:

    • true
    • false

    Default: false

  • high_threshold - High threshold. A frame with a metric value equal to or better than this will never be considered bad

    Syntax: float

    Default: 0.00000000000000000

  • low_threshold - High threshold. A frame with a metric value equal to or worse than this will never be considered bad

    Syntax: float

    Default: 0.00000000000000000

  • naming_type - Scheme for default naming of bad frames image files

    Syntax:

    One of:

    • “frame”
    • “file”
    • “custom”

    Default: “frame”

  • custom_name - Custom name for bad frames image files (supports VQMT environment variables). Only considered if the scheme is CUSTOM

    Syntax: “string”

    Default: “”

• geometry - Settings for geometry correction

  Syntax: {object}

  • enabled - Turn on/off geometry correction. If off, all files mush have the same resolution. Also, geometry correstion can be used to downscale inputs for increasing performance

    Syntax:

    One of:

    • true
    • false

    Default: false

  • size - Target size for resizing or cropping input images

    Syntax:

    One of:

    • “orig”
    • “1/2 orig”
    • “1/4 orig”
    • “min”
    • “max”
    • “custom”

    Default: “orig”

  • width - Target width if size is custom

    Syntax: integer

    Default: 1920

  • height - Target height if size is custom

    Syntax: integer

    Default: 1080

  • strategy - Strategy, that specifies resampling, cropping and ratio preserving

    Syntax:

    One of:

    • “scale”
    • “scale+crop”
    • “crop”

    Default: “scale”

  • algorithm - Image resampling algorithm

    Syntax:

    One of:

    • “nearest”
    • “bilinear”
    • “bicubic”
    • “lanczos”

    Default: “bicubic”

  • antialiasing - Is antialiasing while resampling enabled

    Syntax:

    One of:

    • true
    • false

    Default: false

  • prefer_ffmpeg - Use FFmpeg scaling algorithm if available. If false, Intel algorithm will be always used

    Syntax:

    One of:

    • true
    • false

    Default: true

• mask - Mask specification for indication of the most important areas of the image

  Syntax: {object}

  • mask_type - Type of mask to be aplied

    Syntax: [“string”, “string”]

    Possible types:

    • bin black - binary mask, black for ignored areas
    • bin white - binary mask, white for ignored areas
    • 8bit black - 8-bit mask, black for ignored areas
    • 8bit white - 8-bit mask, white for ignored areas
    • incl-rect <rect spec>, excl-rect <rect spec> - mask consists of rectangle (included or excluded). <rect spec> is ;-separated list of measurements in the order: top, right, bottom, left. Each measurement can be pixel value or percent value (floating point) if followed by ‘%’ sign. A negative value can be used to measure from the opposite side.

    Default: [“no”,”mask”]

  • mask_file - A mask file if maskType considers a file. Could be static image or video file

    Syntax: One of:

    • “string”
    • [“string”, {object}]

    The string is always file name. Possible keys for the object:

    • mode - (string) - explicit mode for reading a file. See also List of open modes. Default: “auto”
    • index - (string) - index mode
    • index_file - (string) - index file. For some index modes will be skipped
    • props - (object) - additional properties, specific for mode and file (see List of open modes)
    • range - (array) - specify array of one element - [N] to skip N frames. Specify array of two elements [N, M] to select frames from N to M inclusive
    • stdin - (bool) - specify true, if the file should be read from stdin
    • offset_type - (string) - if range specified, the way for seeking target frame

    Default: “”

• subsampling - Skipping frames for increasing performance

  Syntax: {object}

  • enabled - Is subsampling enabled. Subsampling could be used for some metrics to skip frames for increasing performance

    Syntax:

    One of:

    • true
    • false

    Default: true

  • do_target_fps - Whether subsampling takes into account target_fps setting. If false, value for every_nth_frame will be taken for subsampling

    Syntax:

    One of:

    • true
    • false

    Default: false

  • every_nth_frame - Number of frames that will be skipped after each processed frame plus one. This value will be taken into account only if do_target_fps is False. Value 1 means no subsampling (each 1st frame is processed)

    Syntax: integer

    Default: 1

  • target_fps - Processing FPS, that VQMT will try to achieve skipping frames

    Syntax: float

    Default: 25.000000000000000

  • enable_seek - Allow VQMT seek in the input files. If false, VQMT will read all intermediate frames

    Syntax:

    One of:

    • true
    • false

    Default: false

• performance - Performance tuning and information

  Syntax: {object}

  • threads_number - Number of threads for performing VQMT tasks. VQMT can spawn additional threads

    Syntax: integer

    Default: 0

  • slots_number - Number of slots that be allocated for each metric to process files simultaneously. `0` - default value. Increasing this value will slightly increase memory consumption, but can improve concurrency

    Syntax: integer

    Default: 0

  • metric_parallelism - How many instances of a single metric can work simultaneously. `0` - default value. Set small values to reduce memory consumption. High metric parallelism is quite useless if you computing many metrics simultaneously

    Syntax: integer

    Default: 0

  • show_performance - Whether information about VQMT performance will be displayed instead of results

    Syntax:

    One of:

    • true
    • false

    Default: false

  • forced_length - The expected number of frames (0 to disable). This parameter affects only progress bar

    Syntax: integer

    Default: 0

• verbosity - Verbocity level. Affects on amount of messages, generated by VQMT

  Syntax:

  One of:

  • “silent”
  • “default”
  • “all”

  Default: “default”

Metric’s reference

Principle quality metrics

PSNR (Peak signal-to-noise ratio)

General info

Algorithm description

Metric depends only on difference of original and distorted, and more preciesly, only on $L_2$-norm of this difference (see MSE). Unlike MSE, metric has logrithmic scale and can be calculated using the following formula:

where MaxErr – maximum possible absolute value of color component (MaxErr=1 in VQMT), w – video width, h – video height.

Total PSNR is aggregated value, that considers all processed frames as a single huge image and then calculates PSNR. Total PSNR takes into account sum suqared distortion on all frames, and doesn’t distinguish situation where all distorion on one frame from situation where it is distributed across all frames. While arithmetic mean aggregated value depends from geometric mean of MSE’s of each frames, Total PSNR depends on arithmetic mean of them.

In MSU VQMT you can calculate PSNR for all YUV and RGB components and for L component of LUV color space. Also, since VQMT 12 you can calculate over all YUV space or all RGB space, achieving a single value for 3 components. PSNR metric is easy and fast to calculate, but sometimes it is not appropriate to human’s perception.

Benchmark

Please note: values can vary depending on system configuration, input format and other factors.

Benchmark on VQMT 14.0 BETA r12792 PRO for Windows.
CPU: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz, 8 cores
GPU: NVIDIA CUDA/GeForce GTX 660 Ti

Benchmark on VQMT 13.1 r12792 PRO for Windows.
CPU: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz, 8 cores
GPU: NVIDIA CUDA/GeForce GTX 660 Ti

Benchmark on VQMT 14.0 BETA r12792 PRO for Linux.
CPU: Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz, 64 cores
GPU: NVIDIA CUDA/TITAN RTX

Netflix VMAF (Video Multimethod Assessment Fusion)

General info

Algorithm description

VMAF is modern reference metric developed by Netflix in cooperation with the University of Southern California. VQMT has full support of VMAF with multiple configuration switches. MSU VQMT support the following VMAF models: VMAF 0.60, VMAF 0.61 (2k, 4k), VMAF 0.62 (2k, 4k), VMAF 0.63 (2k), also, you can compute phone model and elementary features of VMAF. You can use custom model in pkl format with VQMT.

VMAF consist of 4 features (ADM, VIF, Motion, ANSNR) and 35 elementary features, but VMAF models uses only 6 of them: adm2, motion2, vif_scale0, vif_scale1, vif_scale2, vif_scale3. VMAF applies an SVM model to this set of features, which depends on current settings. After applying SVM, the value is clipped to interval 0..100 by default. Motion feature is the only temporal feature, it consider adjacent frames. To calculate VMAF value for current frame it is needed to use the previous frame and the next frame.

VMAF can also compute confidence intervals by applying multiple models and calculating standard deviation of result. VMAF has models, that aimed for 4k and 2k. By default, VQMT will automatically select the correct model by the resolution of input video.

Since VQMT 13 VMAF has a real block-wise visualization, which computes individual VMAF value for each 16x16 block of image. You also can visualize every feature besides motion (ADM, VIF, ANSNR).

Benchmark

Please note: values can vary depending on system configuration, input format and other factors.

Benchmark on VQMT 14.0 BETA r12792 PRO for Windows.
CPU: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz, 8 cores
GPU: NVIDIA CUDA/GeForce GTX 660 Ti

Benchmark on VQMT 13.1 r12792 PRO for Windows.
CPU: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz, 8 cores
GPU: NVIDIA CUDA/GeForce GTX 660 Ti

Benchmark on VQMT 14.0 BETA r12792 PRO for Linux.
CPU: Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz, 64 cores
GPU: NVIDIA CUDA/TITAN RTX

NIQE (Naturalness Image Quality Evaluator)

General info

Algorithm description

NIQE performs feature extraction for every 96x96 block of image on 2 scales. Than it computes correlations of features between all blocks and applies leaned model. For more details, please refer to original paper. Since version 13 VQMT can build a block-wise visualization showing contribution of each block to the final result.

This metric has a special aggregated value NIQE mean, it takes a weighted mean filtering abnormal metric values and taking suspicies values with low weight. You can turn this mode using metric settings. Also, please see our paper.

This metric only applicable to filmed scenes. Metric can produce inadequate result if some graphics is on it (including credits, subtitles, etc.). Please, run NIQE excluding all rendered scenes, they can fatally spoil average value. Also, this metric can produce bad result on scenes containing noisy objects, like sand or grass, however on scenes with big constant areas, like monotonic sky. In common case normal metric results lies in the interval 3..20.

Sometimes, metric shows better result for the compressed image and this correlates with human perception. Compressed image not always is perceived as worse. It can occur for example in case of noisy images (if the noise has non-compression nature). This is only metric in VQMT now that can detect increasing of subjective quality in comparison to original.

Sometimes, codec can allow geometry transformation (like shift of heterogeneous objects in frame), that not critical for subjective perception. Objective-reference metrics are very perceptive to such transformation, and in this cases no-reference metric can show result closer to subjective score.

Benchmark

Please note: values can vary depending on system configuration, input format and other factors.

Benchmark on VQMT 14.0 BETA r12792 PRO for Windows.
CPU: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz, 8 cores
GPU: NVIDIA CUDA/GeForce GTX 660 Ti

Benchmark on VQMT 13.1 r12792 PRO for Windows.
CPU: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz, 8 cores
GPU: NVIDIA CUDA/GeForce GTX 660 Ti

Benchmark on VQMT 14.0 BETA r12792 PRO for Linux.
CPU: Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz, 64 cores
GPU: NVIDIA CUDA/TITAN RTX

SSIM-Family

SSIM (Structural Similarity)

General info

Algorithm description

Main idea of the structure similarity index (SSIM) is to compare distortion of three image components:

• Luminance comparison
• Contrast comparison
• Structure comparison

This algorithm uses some window function $W_{i,j}$, $i,j=0..N$ and performs convolution with this window, defined as follows:

In fast implementation, it uses box window: $W_{i,j}=\frac1{(N+1)^2}$, in other implementations (precise, CUDA, OpenCL, GPU identical) it uses Gaussian window with σ= 1.5, N=10. You can note, that formula above uses negative $U$ indexes, and indexes out of image area. We should define image values outside of it’s edge. In VQMT we spread closest edge pixel to the desired position.

SSIM uses the following convolutions:

And the computed SSIM in each pixel by the following formula:

where $C_1=0.01^2$, $C_2=0.03^2$. Destination metric value is arithmetic mean of SSIM values for each pixel. You also can see SSIM for each individual pixel on visualization.

GPU identical, CUDA, OpenCL implementations should produce very similar result. They use Gaussian window as presice implementation. Value of Presice implementation can differs from these ones.

Benchmark

Please note: values can vary depending on system configuration, input format and other factors.

Benchmark on VQMT 14.0 BETA r12792 PRO for Windows.
CPU: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz, 8 cores
GPU: NVIDIA CUDA/GeForce GTX 660 Ti

Benchmark on VQMT 13.1 r12792 PRO for Windows.
CPU: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz, 8 cores
GPU: NVIDIA CUDA/GeForce GTX 660 Ti

Benchmark on VQMT 14.0 BETA r12792 PRO for Linux.
CPU: Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz, 64 cores
GPU: NVIDIA CUDA/TITAN RTX

MS-SSIM (Multi-Scale Structural Similarity)

General info

Algorithm description

This metric performs SSIM calculation as described in SSIM paragraph for 5 scales of input images. Each next scale divides width and height by 2. The result SSIM values are producted with the following powers: 0.0448, 0.2856, 0.3001, 0.2363, 0.1333.

Benchmark

Please note: values can vary depending on system configuration, input format and other factors.

Benchmark on VQMT 14.0 BETA r12792 PRO for Windows.
CPU: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz, 8 cores
GPU: NVIDIA CUDA/GeForce GTX 660 Ti

Benchmark on VQMT 13.1 r12792 PRO for Windows.
CPU: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz, 8 cores
GPU: NVIDIA CUDA/GeForce GTX 660 Ti

Benchmark on VQMT 14.0 BETA r12792 PRO for Linux.
CPU: Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz, 64 cores
GPU: NVIDIA CUDA/TITAN RTX

3-SSIM (Three-commponent Structural Similarity)

General info

Algorithm description

3-Component SSIM Index based on region division of source frames. There are 3 types of regions – edges, textures and smooth regions. Result metric calculated as weighted average of SSIM metric for those regions. In fact, human eye can see difference more precisely on textured or edge regions than on smooth regions. Division based on gradient magnitude is presented in every pixel of images.

Benchmark

Please note: values can vary depending on system configuration, input format and other factors.

Benchmark on VQMT 14.0 BETA r12792 PRO for Windows.
CPU: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz, 8 cores
GPU: NVIDIA CUDA/GeForce GTX 660 Ti

Benchmark on VQMT 13.1 r12792 PRO for Windows.
CPU: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz, 8 cores
GPU: NVIDIA CUDA/GeForce GTX 660 Ti

Benchmark on VQMT 14.0 BETA r12792 PRO for Linux.
CPU: Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz, 64 cores
GPU: NVIDIA CUDA/TITAN RTX

ST-SSIM (Spatio-Temporal SSIM Index)

General info

Algorithm description

The idea of this algorithm is to use motion-oriented weighted windows for SSIM Index. MSU Motion Estimation algorithm is used to retrieve this information. Based on the ME results, weighting window is constructed for every pixel. This window can use up to 33 consecutive frames (16 + current frame + 16). Then SSIM Index is calculated for every window to take into account temporal distortions as well. In addition, another spooling technique is used in this implementation. We use only lower 6% of metric values for the frame to calculate frame metric value. This causes larger metric values difference for difference files.

Benchmark

Please note: values can vary depending on system configuration, input format and other factors.

Benchmark on VQMT 14.0 BETA r12792 PRO for Windows.
CPU: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz, 8 cores
GPU: NVIDIA CUDA/GeForce GTX 660 Ti

Benchmark on VQMT 13.1 r12792 PRO for Windows.
CPU: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz, 8 cores
GPU: NVIDIA CUDA/GeForce GTX 660 Ti

Benchmark on VQMT 14.0 BETA r12792 PRO for Linux.
CPU: Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz, 64 cores
GPU: NVIDIA CUDA/TITAN RTX

Norm calculation metrics

Delta

General info

Algorithm description

The value of this metric is the mean difference of the color value in the corresponding points of image.

where $m$ – video width, $n$ – video height, image data are in range 0..1. This formula can be rewriten to the following way: $\mbox{mean distorted brightness} − \mbox{mean original brightness}$.

This metric doesn’t show a quality loss, because it can be 0 for completely different images, but you can detect general brightness shifts using this metric if you are sure images have same structure.

Benchmark

Please note: values can vary depending on system configuration, input format and other factors.

Benchmark on VQMT 14.0 BETA r12792 PRO for Windows.
CPU: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz, 8 cores
GPU: NVIDIA CUDA/GeForce GTX 660 Ti

Benchmark on VQMT 13.1 r12792 PRO for Windows.
CPU: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz, 8 cores
GPU: NVIDIA CUDA/GeForce GTX 660 Ti

Benchmark on VQMT 14.0 BETA r12792 PRO for Linux.
CPU: Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz, 64 cores
GPU: NVIDIA CUDA/TITAN RTX

Identity

General info

Algorithm description

Metric cas to modes: binary and pixels. In binary mode only two values are possible: 1 if images are pixel-wise similar, 0 if images have at least 1 different value pixel.

In pixel mode the value is proporsion of similar pixels. 1 means all pixels are similar, 0 means all pixels are dirrefent.

Benchmark

Please note: values can vary depending on system configuration, input format and other factors.

Benchmark on VQMT 14.0 BETA r12792 PRO for Windows.
CPU: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz, 8 cores
GPU: NVIDIA CUDA/GeForce GTX 660 Ti

Benchmark on VQMT 13.1 r12792 PRO for Windows.
CPU: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz, 8 cores
GPU: NVIDIA CUDA/GeForce GTX 660 Ti

Benchmark on VQMT 14.0 BETA r12792 PRO for Linux.
CPU: Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz, 64 cores
GPU: NVIDIA CUDA/TITAN RTX

MSAD (Mean Sum of Absolute Differences)

General info

Algorithm description

This metric has very similar formula to Delta, but has a modulo around the difference:

where $m$ – video width, $n$ – video height, image data are in range 0..1. Metric depends only on difference of original and distorted, it is $L_1$-norm of this difference.

Unlike Delta, this metric will show real difference between images, 0 means completely equivalent images.

Benchmark

Please note: values can vary depending on system configuration, input format and other factors.

Benchmark on VQMT 14.0 BETA r12792 PRO for Windows.
CPU: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz, 8 cores
GPU: NVIDIA CUDA/GeForce GTX 660 Ti

Benchmark on VQMT 13.1 r12792 PRO for Windows.
CPU: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz, 8 cores
GPU: NVIDIA CUDA/GeForce GTX 660 Ti

Benchmark on VQMT 14.0 BETA r12792 PRO for Linux.
CPU: Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz, 64 cores
GPU: NVIDIA CUDA/TITAN RTX

MSE (Mean Squared Error)

General info

Algorithm description

Metric depends only on difference of original and distorted, it is L2-norm of this difference. Metric could be computed using the following formula:

where $m$ – video width, $n$ – video height, image data are in range 0..1.

Benchmark

Please note: values can vary depending on system configuration, input format and other factors.

Benchmark on VQMT 14.0 BETA r12792 PRO for Windows.
CPU: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz, 8 cores
GPU: NVIDIA CUDA/GeForce GTX 660 Ti

Benchmark on VQMT 13.1 r12792 PRO for Windows.
CPU: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz, 8 cores
GPU: NVIDIA CUDA/GeForce GTX 660 Ti

Benchmark on VQMT 14.0 BETA r12792 PRO for Linux.
CPU: Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz, 64 cores
GPU: NVIDIA CUDA/TITAN RTX

Other metrics

VQM (DCT-based Video Quality Metric)

General info