Evaluation methodology of MSU Mobile Video Codec Benchmark

Power Estimation Method

We propose a software-based method for estimating the power consumption of video decoders on various Android devices. The main idea is to retrieve the devices battery level and to access video-decoders using the Android BatteryManager and Media APIs. The formula for computing the relative battery consumption [%] when decoding the entire bitstream and the formula for computing the average decoding speed [frames per second] are as follows:

where the retrieved battery levels are and at times and [seconds], respectively, corresponding to decoding iterations and and frame indices and .

For benchmarking it computes two valuable power metrics. The first is relative battery consumption per hour of playback [% per hour], which lets end-users estimate how long they can watch video in autonomous mode. The second is videodecoding energy per hour, [mA], which helps Android-device manufacturers estimate the power efficiency of hardware video-decoders. Note that includes the energy that the display consumes. We define the display power consumption per hour as [% per hour] and compute it using the same approach by which we track battery-level changes on a device in idle mode with its display on. We calculate as follows:

where is the number of frames in the sequence and is the number of frames per second required for playback. The equation for includes a term compensating the energy that the display consumes for some time when decoding is complete in case of . Note that is required for normal playback. The formula to compute [mA] is the following:

where is a battery capacity [mAh]. Both metrics, and , can help to benchmark mobile devices and videodecoders.

Video Workload

The benchmark tests hardware, software and hybrid video-decoders of six standards (AV1, HEVC, VP9, H.264, VP8, and MPEG-4) available on Android devices. To estimate the power consumption, we selected three video sequences of SD, HD and Full HD resolution and encoded them for each compression standard using FFmpeg. The video sequences were encoded for the streaming case with a fixed-cadence group of picture (GOP), GOP-length equal to two seconds, three references for motion compensation, two B-frames, and maximum bitrate equal to 1.2 of the average bitrate.

The settings restrict to use the Main Profile for H.264 and HEVC with Level 4.2 that supports selected resolutions, frame-rate, and maximum bitrate on all Android 5.0 and higher devices. Our choice of video sequences is based on their distinctive spatial and temporal complexities. Defining the averages of these complexities are the mean spatial perceptual information, , and the mean temporal perceptual information, . Table below lists the , and the average bitrate for each video.

Video Resolution Bitrate (Kbps)
Shakewalk 0.058 124.76 2560
Tractor 0.071 100.57 5120
Zombie 0.073 104.66 12288
16 Jul 2021
See Also
PSNR and SSIM: application areas and critics
Learn about limits and applicability of the most popular metrics
MSU Video Upscalers Benchmark 2021
The most comprehensive comparison of video super resolution (VSR) algorithms by subjective quality
MSU Video Upscalers Benchmark Participants
The list of the participants of the MSU Video Upscalers Benchmark
MSU Video Upscalers Benchmark Methodology
The methodology of the MSU Video Upscalers Benchmark
MSU Super-Resolution for Video Compression Benchmark
Learn about the best SR methods for compressed videos and choose the best model to use with your codec
MSU Super-Resolution for Video Compression Benchmark Participants
The list of participants of MSU Super-Resolution for Video Compression Benchmark
Site structure