Second Annual MSU MPEG-4 AVC/H.264 Video Codec Comparison

For professional users and codec authors

Second H.264/AVC comparison home page!

Part 1. Introduction

Thanks

We would like to express our gratitude to the following companies for providing us with codecs and settings:

• ArcSoft, Inc.
• Ateme
• ATI Technologies Inc.
• Elecard LTD
• Fraunhofer Institute for Integrated Circuits
• Vanguard Software Solutions, Inc.
• x264 Development Team

We would also like to thank these companies for their technical support and help during our tests.

Overview

Codecs

Codec	Developer	Version
1. DivX	DivXNetworks, Inc	6.0
2. ArcSoft H.264	ArcSoft, Inc.	dev. version for 19.09.05
3. Ateme H.264	Ateme	1.2.1.6
4. ATI H.264	ATI Technologies Inc.	3.1.2
5. Elecard H.264	Elecard LTD	dev. version for 09.09.05
6. Franhofer IIS H.264	Fraunhofer Institute for Integrated Circuits	dev. version for 20.09.05
7. VSS H.264	Vanguard Software Solutions, Inc.	3.0.2.7
8. x264	Development group x264	revision 293

  Note: DivX 6.0 is not H.264 codec. It is MPEG-4 ASP codec, it takes part in this testing only as one of the best representative of previous video codec standard MPEG-4

Sequences

Sequence	Number of frames	Frames per second	Spatial dimension and color space
1.       foreman	300	30	352x288(YV12)
2.       susi	374	25	704x576(YV12)
3.       bbc	374	25	704x576(YV12)
4.       battle	1599	24	704x288(YV12)
5.       simpsons	365	24	720x480(YV12)
6.       matrix	239	25	720x416(YV12)
7.       concert	390	25	1664x1088(YV12)

Goal and testing rules

H.264 Codecs Testing Objectives

The main goal of this document is an evaluation of the quality of new H.264 codecs, using objective metrics for comparison. The comparison was done using the settings provided by the developers for each codec.

Testing rules

• Every codecs developers provided settings for both “Best Quality” (high quality compression) and “Best Speed” (better processing speed) presets, except DivX.
• Codec restrictions: Only codecs that supported MPEG-4 AVC/H.264 Main profile and single pass encoding were tested.
• Each codec was tested for speed 3 times, after this the median score (the middle value of the three measurements) was used as the resulting time.
• The following 10 bitrates were used (in Kilobits per second) 100, 225, 340, 460, 700, 938, 1140, 1340, 1840, 2340.
• During the testing three types of video sequences were used:
  • Source sequences (.yuv extension) in the YUV 4:2:0 format.
  • (.avi extension) sequences in the YUV 4:2:0 format.

  These sequences were made in the following way: Sequence was opened in VirtualDub using AviSynth script and, after that, it was saved in AVI file format. VirtualDub parameters “Color depth - Decompression Format” and “Color depth - Output format for compressor” were set to “Autoselect” and “Same as decompression format” respectively. AviSynth script for sequence opening:

    RawSource("\<source_file\>.yuv", \<wigth\>, \<height\>, "I420")
    AssumeFPS(\<source_fps\*1000\>, 1000)
  

  After that in obtained YUV file fourcc was manually changed from “DIB “ to “YV12” using HexEditor from VirtualDub. This was required for correct work of Ateme codec.

  • *.yuv sequences with duplicated last frame. These sequences were used for testing x264 codec (see its description). Written for this case, special utility was used for generating such sequences.
• In this testing VirtualDub version 1.6.10 was used.
• In this testing AviSynth version 2.55 was used.
• For all metrics’ measurements in this test special build of MSU Video Quality Measure was used.
• Two computers with following configuration were used for encoding:
  • Processor: Pentium 4, 2.8 GHz with Hyper Threading
  • Operating system: Windows 2000 Pro, SP4
  • Memory: 1Gb
  • Video Accelerator: ATI Radeon 9600 Series
  • Hard Drives: ATA100 2x80Gb

Metrics Used in Comparison

During testing following metrics were measured: PSNR (Y, U, V, L, R, G, B components) SSIM (Y component) VQM (Y component) Blurring (Y, U, V components) Blocking (Y, U, V components) Detailed description of these metrics including VQM and SSIM algorithms could be found here: /vqmt/info.html

Number of graphs

Graph type	Total number of graph	Inserted into this document
Average metric	196	24 (12%)
Relative average	196	7 (3.5%)
Per frame metric	1414	13 (0.9%)
Bitrate Handling	14	14 (100%)
Speed/Quality	1960	8 (0.4%)
Speed	16	16 (100%)
Total	3796	82 (2%)

Sequences

Foreman

Sequence title	foreman
Resolution	352x288
Number of frames	300
Color space	YV12
Frames per second	30
Source	Uncompressed (standard sequence), progressive

Picture 1. Frame 77

Picture 2. Frame 258

This is one of the most famous sequences. It represents a face with very rich mimic. Motion is not very intensive here, but on the other hand it is disordered, not forward. Intricate character of motion creates problems for the motion compensation process. In addition camera is shaking that makes the image unsteady. In the end of the sequence camera suddenly turns to the building site and there follows an almost motionless scene. So this sequence also shows codec’s behavior on a static scene after intensive motion.

Susi

Sequence title	susi
Resolution	704x576
Number of frames	374
Color space	YV12
Frames per second	25
Source	MPEG-2 (40Mbit), Smart Deinterlace

Picture 3. Frame 193

This sequence is characterized by high-level noise and slow motion. In its first part the scene is almost static (the girl only blinks), then there is some motion (she abruptly moves her head) and then the scene becomes almost static again. Noise is suppressed on every second frame due to the B-frames option in MPEG-2 codec.

BBC

Sequence title	bbc
Resolution	704x576
Number of frames	374
Color space	YV12
Frames per second	25
Source	Uncompressed (standard sequence), Smart Deinterlace

Picture 4. Frame 185	Picture 5. Frame 258

This sequence is characterized by pronounced rotary motion. It contains a rotating striped drum with different pictures and photos on it. Quality of the compressed sequence can be evaluated by observing details on these images.

Battle

Sequence title	battle
Resolution	704x288
Number of frames	1599
Color space	YV12
Frames per second	24
Source	MPEG-2 (DVD), FlaskMPEG deinterlace

Picture 6. Frame 839

This sequence is a fragment of the “Terminator-2” movie, which represents its very beginning. In terms of compression this sequence is the most difficult one among all other sequences that took part in the testing. That is because of three main reasons: constant brightness changes (explosions and laser flashes, see the picture above), very quick motion and frequent changes of the scene that make codecs often compress frames as I-frames.

Simpsons

Sequence title	simpsons
Resolution	720x480
Number of frames	365
Color space	YV12
Frames per second	24
Source	MPEG-2 (DVD), progressive

Picture 7. Frame 50

This sequence is a fragment of “Simpsons” cartoon film. This is a classical representative of cartoon films: sketchy movement, great number of monochrome regions with abrupt junctions between them. Previously this sequence was compressed in MPEG-2 with rather low bitrate.

Matrix

Sequence title	matrix
Resolution	720x416
Number of frames	239
Color space	YV12
Frames per second	25
Source	MPEG-2 (DVD), Smart Deinterlace

Picture 8. Frame 226

This sequence is a fragment of “Matrix” movie. Relatively simple movement, quite dim colors and small resolution allows codec to treat this sequence in rather simple way.

Concert

Sequence title	concert
Resolution	1664x1088
Number of frames	390
Color space	YV12
Frames per second	25
Source	MPEG-2 (DVD), Smart Deinterlace

Picture 9. Frame 128

This sequence is a part of HDTV broadcast of symphonic orchestra concert. Sequence’s spatial resolution is very high. At the same time motion is rather simple and sometimes it completely disappears. There are two scene changes in this sequence.

Codecs

DivX 6.0

• DivX 6.0 is not H.264 codec. It is MPEG-4 ASP codec, it takes part in this testing only as one of the best representative of previous video codec standard MPEG-4
• This is a VfW (Video for Windows) codec.
• Compression was performed using VirtualDub 1.6.10 video processing program.
• Evaluation version of codec works for 6 months.
• There were no presets from developers. All tests were performed using “Home Theater Profile”. For different test presets only Codec Performance settings were changed:
  • For “Best Quality” mode preset “Insane quality” was used
  • For “Best Speed” mode preset “Balanced” was used

Remarks:

• Codec shifts source video sequence by one frame while using its own decoder. With other decoders (specifically, XviD) this effect disappears. For removing such shift next AviSynth script was used:

    clip = AVISource("source.avi")
    clip = clip.DeleteFrame(0)
    clip = clip.DuplicateFrame(clip.FrameCount()-1)
    return clip
  

• Codec generates an error on “concert” sequence’s compression.

However, despite these drawbacks, codec significantly improved compression quality in comparison with previous version. On the next graph there is a comparison between different versions of DivX (metric is Y-PSNR):

Picture 11. DivX 6.0, DivX 5.21 and DivX 3.22 comparison

 

ArcSoft H.264

• Console encoding program.
• DirectShow filters were used for decoding.
• Codec (encoder and decoder) and presets were provided by ArcSoft, Inc company specially for this test.

Remarks:

• Codec works without remarks.

 

Ateme H.264

Picture 12. Ateme H.264

• Console encoding program.
• Reference decoder JM9.8 was used for decoding.
• Codec and presets were provided by Ateme company specially for testing.

Remarks:

• Codec stores compressed sequence in MPEG4 container. Program mp4box version 0.4.0 was used for extracting H.264 stream from it.

 

ATI H.264

• Console encoding program.
• Reference decoder JM9.8 was used for decoding.
• Codec and presets were provided by ATI Technologies Inc. company specially for this test.

Remarks:

• Codec has only “Best Speed” preset.

 

Elecard H.264

• Console encoding program.
• Console decoder provided by developers was used for decoding.
• Codec (encoder and decoder) and presets were provided by Elecard LTD company specially for this test.

Remarks:

• Codec works without remarks.

 

Fraunhofer IIS H.264

Picture 13. Fraunhofer IIS H.264

• Console encoding program.
• Reference decoder JM9.8 was used for decoding.
• Codec and presets were provided by Fraunhofer Institute for Integrated Circuits IIS specially for this test.

Remarks:

• Codec works without remarks.

 

VSS H.264

Picture 14. VSS H.264

• Console encoding program.
• Console decoder was used for decoding.
• Codec (encoder and decoder) and presets were provided by Vanguard Software Solutions, Inc. company specially for this test.

Remarks:

• Codec generates an error while compressing “foreman” sequence.

 

x264

Picture 15. x264

• Console encoding program.
• Program ffmpeg 2005.09.09 was used for decoding.
• This codec is an Open Source project.
• Optimal settings were provided by codec’s Development Team.

Remarks:

• During decoding ffmpeg program does not write last frame of video sequence. Therefore we used sequences with duplicated last frame for this codec.