Second Annual MSU MPEG-4 AVC/H.264 Video Codec Comparison

For professional users and codec authors

Second H.264/AVC comparison home page!

Part 4. Additional information and conclusions

2-pass mode and High profile

Additionally to one-pass mode in Main profile of standard H.264 we tested two-pass mode and compression using possibilities of High profile. Only x264, Elecard, Ateme and Fraunhofer IIS codecs took part in this test. This test was held on “foreman” and “battle” sequences and without speed measuring. As it is seen on picture below, it is difficult to analyze results using PSNR/Bitrate graphs. That is why we use Delta Y-PSNR graphs instead of absolute Y-PSNR graphs in this part. In all graphs reference is Main Profile one pass results for current codec.

 

Sequence “foreman”, preset “Best quality”

 

 

 

 

 

 

Sequence “battle”, preset “Best quality”

 

 

 

 

 

Note:

Fraunhofer IIS H.264 codec showed some not very adequate results during this part of testing because of presets that were provided for us: Main profile with two passes has quality parameter equal to 4 (good quality) Main profile with one pass and High profile have quality parameter equal to 6 (best quality)

Per frame comparison

In addition to average metrics’ values we measured metrics’ values for separate frames. Basing on this data, Rate Control of each codec could be analyzed. As a rule, on such graphs it is clearly seen whether codec uses B-frames or not (“striation” of graph), location of I-frames (abrupt peaks or collapses on graph), scene changing reaction, quality of R-D models, etc.

On the next graphs there are frames on abscissa axis and bit rates on axis of ordinates (low bit rates are below). Color shows metric’s value: the more red, the lesser is value. Thus, for Y-PSNR metric red color means bad quality, blue - good quality. All graphs of one sequence have the same colors (every color corresponds to only one value of metric on all graphs).

Analyzing these graphics we should take into account bit rate handling of codecs since these graphs do not consider it.

Sequence “foreman”, Y-PSNR, Preset “Best quality”

 

 

 

 

 

 

   

Sequence “bbc”, Y-PSNR, Preset “Best quality”

On this graph compression quality does not significantly depend from bit rate, because DivX codec does not maintain low bit rates well, and works badly on high ones.

 

 

 

 

 

Remarks: These graphs show that x264, Elecard, Ateme and VSS codecs use B-frames. DivX and Elecard codecs have very interesting B-frames placement algorithm. They change positions of each type of frames depending on bit rate value. This is shown on graphs as vertical wavy lines.

“bbc” sequence clearly shows problems in bit rate control of ArcSoft codec. It is represented as presence of “holes” on average bit rates. Considering complex circular motion in this sequence, one could suppose presence of problems with motion estimation in this codec.

Large quality overstating of I-frames in x264 codec (it is also typical for Ateme and Fraunhofer IIS codecs, but at smaller degree) may be caused by inaccurate prediction of current frame type. It is represented as distinct blue lines that appear several times in sequence.

Visual comparison

Even in spite of appearance of new metrics, that more adequately reflect human perception of video, subjective video tests still do not lose their importance.

We set ourselves a problem to compare video codecs using only existing objective metrics. But in addition to various graphs that impartially evaluate quality of obtained results, below we give several frames from “bbc” and “battle” sequences, compressed with different codecs.

Sequence “bbc”, frame 170, bitrate 1140 kbps

 

Sequence “bbc”, frame 250, bitrate 1140 kbps

 

Sequence “battle”, frame 527, bitrate 700 kbps

 

 

 

 

 

 

 

Sequence “battle”, frame 527, bitrate 700 kbps, Y-PSNR

Informal codec comparison

  As it is clearly seen on the graphs, codecs show significantly different results on different sequences. Selected sequences have different types in terms of motion and noise, and this allows checking which codecs are tuned to all types of sequences, and which one - to only one. But also it is very interesting to evaluate codec behavior on all test data.

We considered averaging of results between different sequences as incorrect, and used a method of assigning points to codecs for different measurements. Points are united in arbitrary informal rating in the end.

We tried to reflect real situation on test data as much as possible. We are still developing methodology of our informal comparison. Therefore it is better to consider following results as subjective opinion of comparison authors.

Informal comparison rules

Separate comparisons were carried out for “Best Quality” and “Best Speed” presets. Several graphs were chosen for each preset, on which comparison was held. Obtained marks were summed with some predefined weights.

Codecs received points in a following way:

• If codec is one of the leaders on current graph then it received 3 points.
• If codec is trailing on current graph then it received 1 point.
• Otherwise it is received 2 points.

Following graphs were used for “Best Quality” preset with corresponding coefficients:

• Y-PSNR graphs, coefficient 4.
• U-PSNR graphs, coefficient 1.
• V-PSNR graphs, coefficient 1.
• SSIM graphs, coefficient 1.
• VQM graphs, coefficient 1.
• Bit rate handling graphs, coefficient 3 (denoted in tables as “BH”).
• Graphs of absolute speed, coefficient 1.

Following graphs were used for “Best Speed” preset with corresponding coefficients:

• Y-PSNR graphs, coefficient 1.
• Bit rate handling graphs, coefficient 2 (denoted in tables as “BH”).
• Graphs of absolute speed, coefficient 4.

Average values were computed as arithmetic average of all successfully encoded sequences (i.e. codecs’ errors were not taken into account).

Informal comparison results

Preset “Best quality

Sequence “foreman”

  Sequence “susi”

  Sequence “bbc”

  Sequence “battle”

  Sequence “simpsons”

  Sequence “matrix”

  Sequence “concert”

  Results

Preset “Best speed”

Sequence “foreman”

 

Sequence “susi”

 

Sequence “bbc”

 

Sequence “battle”

 

Sequence “simpsons’

 

Sequence “matrix”

 

Sequence “concert”

 

Results

 

General conclusions

Comparing to previous year comparison, the level of H.264 standard codecs has increased significantly. One year ago best codecs of old MPEG4-ASP standard were comparable to best codecs of H.264 standard, now this situation greatly changed. DivX codec is inferior to most codecs of new standard, even despite considerable growth of its quality comparing to previous version.

It is rather difficult to choose absolute leader by quality among codecs of new standard. x264 and Ateme codecs showed approximately equal best results.

Obvious leader by speed is codec from ATI company. It was significantly faster than the others while not always showing worst results.