Evaluation methodology of MSU Mobile Video Codec Benchmark
Power Estimation Method
We propose a software-based method for estimating the power consumption of video decoders on various Android devices. The main idea is to retrieve the devices battery level and to access video-decoders using the Android BatteryManager and Media APIs. The formula for computing the relative battery consumption [%] when decoding the entire bitstream and the formula for computing the average decoding speed [frames per second] are as follows:
where the retrieved battery levels are and at times and [seconds], respectively, corresponding to decoding iterations and and frame indices and .
For benchmarking it computes two valuable power metrics. The first is relative battery consumption per hour of playback [% per hour], which lets end-users estimate how long they can watch video in autonomous mode. The second is videodecoding energy per hour, [mA], which helps Android-device manufacturers estimate the power efficiency of hardware video-decoders. Note that includes the energy that the display consumes. We define the display power consumption per hour as [% per hour] and compute it using the same approach by which we track battery-level changes on a device in idle mode with its display on. We calculate as follows:
where is the number of frames in the sequence and is the number of frames per second required for playback. The equation for includes a term compensating the energy that the display consumes for some time when decoding is complete in case of . Note that is required for normal playback. The formula to compute [mA] is the following:
where is a battery capacity [mAh]. Both metrics, and , can help to benchmark mobile devices and videodecoders.
Video Workload
The benchmark tests hardware, software and hybrid video-decoders of six standards (AV1, HEVC, VP9, H.264, VP8, and MPEG-4) available on Android devices. To estimate the power consumption, we selected three video sequences of SD, HD and Full HD resolution and encoded them for each compression standard using FFmpeg. The video sequences were encoded for the streaming case with a fixed-cadence group of picture (GOP), GOP-length equal to two seconds, three references for motion compensation, two B-frames, and maximum bitrate equal to 1.2 of the average bitrate.
The settings restrict to use the Main Profile for H.264 and HEVC with Level 4.2 that supports selected resolutions, frame-rate, and maximum bitrate on all Android 5.0 and higher devices. Our choice of video sequences is based on their distinctive spatial and temporal complexities. Defining the averages of these complexities are the mean spatial perceptual information, , and the mean temporal perceptual information, . Table below lists the , and the average bitrate for each video.
Video | Resolution | Bitrate (Kbps) | ||
---|---|---|---|---|
Shakewalk | 0.058 | 124.76 | 2560 | |
Tractor | 0.071 | 100.57 | 5120 | |
Zombie | 0.073 | 104.66 | 12288 |
-
MSU Benchmark Collection
- Super-Resolution for Video Compression Benchmark
- Video Colorization Benchmark
- Defenses for Image Quality Metrics Benchmark
- Learning-Based Image Compression Benchmark
- Super-Resolution Quality Metrics Benchmark
- Video Saliency Prediction Benchmark
- Metrics Robustness Benchmark
- Video Upscalers Benchmark
- Video Deblurring Benchmark
- Video Frame Interpolation Benchmark
- HDR Video Reconstruction Benchmark
- No-Reference Video Quality Metrics Benchmark
- Full-Reference Video Quality Metrics Benchmark
- Video Alignment and Retrieval Benchmark
- Mobile Video Codecs Benchmark
- Video Super-Resolution Benchmark
- Shot Boundary Detection Benchmark
- The VideoMatting Project
- Video Completion
- Codecs Comparisons & Optimization
- VQMT
- MSU Datasets Collection
- Metrics Research
- Video Quality Measurement Tool 3D
- Video Filters
- Other Projects