Sixth MPEG-4 AVC/H.264 Video Codecs Comparison - Short Version

History of versions

25th May 2010: Public report is released

16th June 2010: Appendix with VP8 encoder analysis is released

Different Versions of Report

There are four different versions of H.264 Comparison 2010 report:

Here is the comparison of the versions:

Short Version Pro Version (Movies) Pro Version (HDTV) Pro Version (Enterprise)
Objective Metrics (SSIM) YES YES YES YES
Additional Metrics (PSNR) NO For all results SSIM and PSNR metrics are used For all results SSIM and PSNR metrics are used For all results SSIM and PSNR metrics are used
Movie use case Only Normal preset High Speed, Normal, High Quality presets NO High Speed, Normal, High Quality presets
HDTV use case NO NO High Speed, Normal, High Quality presets High Speed, Normal, High Quality presets
Subjective Comparison NO NO NO SAMVIQ analysis including psycho-visual enhancements
Additional Analysis NO NO NO Synthetic Motion Analyzer
Additional Analysis NO NO NO Distortion in Tail Area Analyzer
Additional Analysis NO NO NO Spatially Variable Noise Analyzer
Number of pages 25 116 114 177
Number of figures 50 131 131 226
Prices Free $349 $349 $699
Purchase Download pdf Buy Buy Buy
Hint: You can remove "Extended download" service while purchasing to save money.
We can help you to analyze your codec

Pro versions of report will be available immediately after report purchasing.

Html-Version of Report

Video Codecs that Were Tested

Overview

Sequences

Table 1. Summary of video sequences
Sequence Number of frames Frame rate Resolution
Ice Age 2014 24 720x480
Indiana Jones 5000 30 704x288
State Enemy 6500 24 720x304
Up 1920 24 720x480
Amazon 1200 24 1280x720
Iron Man 600 24 1920x1080
Mobile Calendar 504 50 1280x720
Troy 300 24 1920x1072

Brief descriptions of the sequences used in our comparison are given in Table 1. More detailed descriptions of these sequences can be found in Appendix 4. Test Set of Video Sequences

Codecs

Table 2. Short codec descriptions
Codec Developer Version
DivX AVC/H.264 Video Encoder DivX, Inc. version 1.1.1.6
Elecard AVC Video Encoder 8-bit edition Elecard Ltd 2.1.022202.091207
Intel® MediaSDK AVC/H.264 transcoder sample application Intel Corp. 1.10.1.15
MainConcept AVC/H.264 Video Encoder Console Application MainConcept GmbH 8.5.0.12837
Microsoft Expression Encoder 3 Microsoft Corp. 3.0.1332.0
Theora encoder Xiph.Org. Provided by developers
x264 x264 Development Team x264 core:85 r1442M 781d300
XviD raw mpeg4 bitstream encoder XviD Development Team xvid-1.3.0-dev

Brief descriptions of the codecs used in our comparison are given in Table 2. XviD was used as a good quality MPEG-4 ASP reference codec for comparison purposes. Detailed descriptions of all codecs used in our comparison can be found in Appendix 5. Tested Codecs and Presets.

Every codec developer could take private participation in this comparison to test and analyze its codec.

Objectives and Testing Rules

H.264 Codec Testing Objectives

The main goal of this report is the presentation of a comparative evaluation of the quality of new H.264 codecs using objective measures of assessment. The comparison was done using settings provided by the developers of each codec.

The main task of the comparison is to analyze different H.264 encoders for the task of transcoding video—e.g., compressing video for personal use. Speed requirements are given for a sufficiently fast PC; fast presets are analogous to real-time encoding for a typical home-use PC.

Testing Rules

More detailed information about these measures may be found on the Internet at the Quality Measure page.

Results for Movies

Full results for Movies could be found in professional versions of this report:
H.264 Comparison Report Pro 2010 - Movie edition
H.264 Comparison Report Pro 2010 - Enterprise edition

This version contains only Y-SSIM results only for Normal preset.

RD Curves

The Normal preset results for each sequence are presented in next figures. The results do not depend strongly on the metric used. SSIM metric: The leader is x264; MainConcept placed second, and DivX H.264 placed third. For the “Ice Age” and “Up” sequences, the MediaSDK and Elecard encoders exhibited similar results. For the “Indiana Jones” and “State Enemy” sequences, the XviD encoder placed fourth.

Encoding Speed

Absolute speed results are presented in next figures. All the encoders except Microsoft Expression and Theora have a similar growth rate for encoding time versus increasing bitrate. Elecard is the fastest encoder. The encoding speed of the Microsoft Expression and Theora encoders exhibits almost no dependency on bitrate, and Microsoft Expression’s encoding speed decreases at 1,200 kbps for the “Ice Age” sequence. XviD’s encoding speed exhibits little dependency on bitrate for the “Up” sequence.

Speed/Quality tradeoff

Detailed descriptions of the speed/quality trade-off graphs can be found in Appendix 7. Sometimes, codec results are not present in the particular graph owing to the codec’s extremely poor performance. The codec’s RD curve has no intersection with the reference’s RD curve.

The speed/quality trade-off graphs simultaneously show relative quality and encoding speed for the encoders tested in this comparison. XviD is the reference codec, for which both quality and speed are normalized to unity for all of the graphs. The terms “better” and “worse” are used to compare codecs in the same manner as in previous portions of this comparison.

Please note that the method of averaging among all sequences assumes that all codecs produced results for each sequence. When this is not the case, only existing results are taken into account.

Next figures show results for the Normal preset. The results differ depending on the chosen metric.

Y-SSIM: The three best codecs (no codec performs faster with higher quality) in terms of speed and quality are XviD, Elecard and x264 on average, except for the “Ice Age” and “Indiana Jones” sequences. The x264 encoder is better on average than DivX H.264 and MainConcept.

Almost all encoders except Microsoft Expression exhibited better results on average than did Theora.

Bitrate Handling

Encoders with Normal presets, except the Microsoft Expression encoder, demonstrate good bitrate handling for all sequences. For the “Up” sequence, XviD showed less than stellar results, especially at high bitrates. For all sequences, MainConcept, x264, Elecard and DivX H.264 showed the best results compared with other encoders. DivX H.264’s bitrate decreases at 1,500 kbps. MediaSDK slightly decreases bitrates, but the results are stable. Theora has a good bitrate handling mechanism but is not very stable.

Relative Quality Analysis

Next table shows relative bitrates for a fixed-quality output for all codecs. Note that these tables do not include information about the speed of the encoder.

Note that each number in the tables below corresponds to some range of bitrates (see Appendix 7 for more details). Unfortunately, these ranges can differ significantly because of differences in the quality of compared encoders. This situation can lead to some inadequate results when three or more codecs are compared. This comparison technique will be improved in the future.

The leaders are x264 and MainConcept, depending on the quality metric (the bitrate difference is 18% for a fixed quality).

Table 3. Average bitrate ratio for a fixed quality—usage area “Movies", Normal preset, Y-SSIM metric
DivX H.264 Elecard Expression MediaSDK MainConcept Theora x264 XviD
DivX H.264 100% 124% 127% 118% 87% 138% 76% 128%
Elecard 81% 100% 103% 95% 70% 115% 61% 104%
Expression 79% 97% 100% 93% 69% 114% 61% 101%
Media SDK 84% 105% 107% 100% 74% 120% 64% 108%
MainConcept 115% 142% 144% 136% 100% 157% 88% 145%
Theora 72% 87% 88% 84% 64% 100% 57% 88%
x264 131% 163% 164% 155% 114% 176% 100% 164%
XviD 78% 96% 99% 93% 69% 113% 61% 100%

Next figure depicts the data from the table above. Each line in the figures corresponds to one codec. Values on the vertical axis are the average relative bitrates compared with the codecs along the horizontal axis. A lower bitrate indicates better relative results.

Results for HDTV

Full results for HDTV could be found in professional versions of this report:
H.264 Comparison Report Pro 2010 - HDTV edition
H.264 Comparison Report Pro 2010 - Enterprise edition

Speed/Quality tradeoff

Detailed descriptions of the speed/quality trade-off graphs can be found in Appendix 7. Sometimes, codec results are not present in the particular graph owing to the codec’s extremely poor performance. The codec’s RD curve has no intersection with the reference’s RD curve.

The speed/quality trade-off graphs simultaneously show relative quality and encoding speed for the encoders tested in this comparison. XviD is the reference codec, for which both quality and speed are normalized to unity for all of the graphs. The terms “better” and “worse” are used to compare codecs in the same manner as in previous portions of this comparison.

Please note that the method of averaging among all sequences assumes that all codecs produced results for each sequence. When this is not the case, only existing results are taken into account.

Conclusions

Movies

The leading encoders in this usage area are MainConcept and x264. The quality of the Theora encoder is rather low.

High Speed Preset

The x264 encoder demonstrates better quality on average, and MainConcept shows slightly lower quality. These codecs’ bitrate handling algorithm is acceptable for this usage area. The MediaSDK codec places third, along with the dicas encoder (MediaSDK has slightly higher quality, but its encoding speed is 25% lower). The top four codecs for this preset are the following:

  1. x264
  2. MainConcept
  3. MediaSDK

Normal Preset

The results for the Normal preset differ from those for the High Speed presets only in third place. The x264 encoder demonstrates better quality on average, and MainConcept shows slightly lower quality. The DivX H.264 encoder holds third place. The top three codecs for this preset are the following:

  1. x264
  2. MainConcept
  3. DivX H.264

High Quality Preset

The results for this preset are similar to those of the Normal preset. The leaders are the x264 and MainConcept codecs. DivX H.264 takes third place. The top three codecs for this preset are the following:

  1. x264
  2. MainConcept
  3. DivX H.264

*The encoding speed of these encoders is not in the requirements range owing to our chosen presets or to a lack of options
**The MediaSDK encoder does not meet the requirements for High Speed presets–specifically, no B-frames and exactly one reference frame

HDTV

The leaders in the HDTV area are x264, DivX H.264 and MainConcept. The XviD encoder trails all other H.264 encoders. Theora demonstrates rather poor results compared with other encoders.

High Speed Preset

The x264 encoder demonstrates better quality on average, and MainConcept shows slightly lower quality. The top three codecs for this preset are the following:

  1. x264
  2. DivX H.264 (this preset does not meet speed requirements)
  3. MainConcept

Normal Preset

The results for the Normal preset differ from those of the High Speed presets. MainConcept shows the best results. The difference between best codecs are 2-4% of bitrate for the same quality. The top three codecs for this preset are the following:

  1. MainConcept
  2. x264
  3. DivX H.264

High Quality Preset

The results for the High Quality preset are very interesting: x264 held first place, MainConcept was in second place, and two codecs (DivX H.264 and Elecard) shared third place (their quality results are very close, but DivX H.264 is 1.2 times faster than Elecard). The top four codecs for this preset are the following

  1. x264
  2. MainConcept
  3. DivX H.264 and Elecard
Average bitrate for HDTV for all presets

*The encoding speed of this encoder is not in the requirements range owing to our chosen presets or to a lack of options
**The DivX H.264 High Speed preset for HDTV does not meet the speed requirements

Overall Conclusions

Overall, the leader in this comparison is x264, followed by MainConcept and DivX H.264. The Theora encoder demonstrates the worst results among all codecs tested.

Average bitrate for Movies and HDTV for all presets

*The encoding speed of this codec does not fall in the required range owing to our chosen presets or to a lack of options.

The overall ranking of the codecs tested in this comparison is as follows:

  1. x264
  2. MainConcept
  3. DivX H.264
  4. MediaSDK
  5. Elecard
  6. XviD
  7. Theora

Microsoft Expression Encoder 3 could not be placed in this list because of its much longer encoding time compared with other encoders (except Theora).

Using the standard Theora interface, we could not find any multithreading options; Theora works only in single-thread mode. The leader in this comparison is x264—its quality difference (according to the SSIM metric) could be explained by the special encoding option (“–tune ssim”). Interestingly, using the PSNR metric for MainConcept yielded results comparable with or better than those of x264. This means that no encoder can achieve the best results for both SSIM and PSNR when using the same parameters.

The difference between the MainConcept and DivX H.264 encoders is not overly significant, so these encoders tied for second in this comparison. The developers of the Elecard encoder do not provide a High Speed preset, so its ranking is based solely on the results for the Normal and High Quality presets. The dicas encoder holds third or fourth place for some sequences.

This rank is based only on the encoders’ quality results. Encoding speed is not considered here.

Codec Conclusion

DivX H.264 — quite balanced encoder with not very big number of parameters, this fact could be comfortable for users. This encoder is designed as a free sample application for DivX Plus HD compliant video encoding, and is a feature-constrained, for-purpose application.

Elecard — very fast codec with good encoding quality and very flexible settings. Many adjustable encoding settings are provided. This encoder has a very good bitrate handling mechanism (especially for the “Movies” usage area).

Microsoft Expression Encoder — the low encoding speed could result from the GUI loading every time a video is encoded. Unfortunately, we could not automate the encoder to eliminate this GUI loading.

MediaSDK — balanced encoder with a limited number of parameters but several use cases (speed/quality presets), making the encoder very comfortable for users.

MainConcept — one of the best codecs by encoding quality; has many encoding settings that can be adjusted. This encoder has a very good bitrate handling mechanism.

Theora — not an H.264 codec. Using our testing methodology (objective metrics) and test set, this encoder yields lower quality than do H.264 codecs and even MPEG-4 ASP.

x264 — one of the best codecs by encoding quality; has very user-friendly predefined presets, as well as many adjustable encoding settings.

XviD — an MPEG-4 ASP codec; its quality could be very close to or even higher than that of some commercial H.264 standard implementations, especially for encoding “Movie” sequences, but not for “HDTV” sequences.

Professional Versions of Comparison Report

H.264 Comparison Report Pro 2010 versions contain:

Acknowledgments

The Graphics & Media Lab Video Group would like to express its gratitude to the following companies for providing the codecs and settings used in this report:

The Video Group would also like to thank these companies for their help and technical support during the tests.

Thanks

Special thanks to the following contributors of our previous comparisons

Google Intel AMD NVidia
ATI Adobe ISPhone dicas
KDDI R&D labs Dolby Tata Elxsi Octasic
Qualcomm Voceweb Elgato

Appendixes

Appendix 1. Subjective Comparison and Psycho-visual Enhancements

Appendix 2. x264 Over-years Comparison

Appendix 3. Theora vs. x264 Single-thread Comparison

Appendix 4. Test Set of Video Sequences

Appendix 5. Tested Codecs and Presets

Appendix 6. Figures Explanation

Appendix 7. Objective Quality Metrics Description

VP8, x264 and XviD comparison

Codec Analysis and Tuning for Codec Developers and Codec Users

Computer Graphics and Multimedia Laboratory of Moscow State University:

Strong and Weak Points of Your Codec

Independent Codec Estimation Comparing to Other Codecs for Different Use-cases

Encoder Features Implementation Optimality Analysis

We perform encoder features effectiveness (speed/quality trade-off) analysis that could lead up to 30% increase in the speed/quality characteristics of your codec. We can help you to tune your codec and find best encoding parameters.

Contacts

E-mail: videocodec-testing@graphics.cs.msu.ru

08 Mar 2013
See Also
Video Colorization Benchmark
Explore the best video colorization algorithms
Call for HEVC codecs 2019
Fourteen modern video codec comparison
HEVC Video Codecs Comparison 2018 (Thirteen MSU Video Codec Comparison)
13th MSU video codecs comparison
HEVC Video Codecs Comparison 2017 (Twelfth MSU Video Codec Comparison)
12th MSU video codecs comparison
MSU Video Codec Comparisons (6 test of lossless, MPEG-4 and MPEG-4 AVC)
Call for HEVC codecs 2018
Site structure