There are lots of articles talking about how to do video codec comparison. I’m writing my personal experience here to make a record for myself and hope it is helpful to others.
As long as your business has video involved, I believe a codec is a must-have tool to facilitate the various operations. As far as I know, CDN or bandwidth cost is usually the second largest expense for a video monitization company, just below copyright cost. A good video codec, especially encoder, can save millions of dollars for those companies, which is the major reason for them to optimize it. The performance of different video codecs are differentiated by the compatible compression standard (MPEG-1. MPEG-2, H.264/AVC, H.265/HEVC, VP8, VP9, etc.), as well as the specific deployed techniques (advanced rate control scheme, adaptive quantization technique, psychovisual enhancement tool etc.) and corresponding implementations. Thus, comparing different video codecs is definitely a non-trivial task and require lots of efforts, especially for fair comparison.
The comparison is not only among video codecs, but also among configurations for the same encoder, in which case, fair comparison is more feasible. The comparison can be either subjectively (expert’s eyes) or objectively (PSNR, SSIM, or SSIMplus etc). I remember when I was a research intern at RIM (now Blackberry), I need implemented new ideas and tested them by compare with the original setting iteratively. Most of comparison are objective because PSNR values or RD curve are still the most important criteria that people are looking at. Subjective test is usually employed as the final verification, also due to its time-consuming problem.
During my PhD study, both subjective and objective experiment are conducted to solve a quality assessment (QA) problem. Subjective user study was used to collect mean opinion score (MOS) which can be used as benchmark for objective QA algorithm development. BT.500 is the most cited document, I guess, for subjective study instruction. But the major problems are time and labor comsuming, and limited test samples.
Objective methods can run 24/7 and automatically report comparison results, but the accuracy is the major concern. PSNR had been widely considered as a poor quality indicator. SSIM is better but not good enough. SSIMplus seems to be a much better candidate who has potential to be industry standard because of its exclusive features (accurate, real-time speed, find granularity, cross-content, and cross-resolution)
The basic procedure of compare video codecs are
(1) encode test sequence with original configuration or codec A;
(2) encode test sequence again with new configuration or codec B;
(3) repeat (1) and (2) four times with different quality/bitrate factor each time;
(4) If subjective test is used, then draw bitrate vs. MOS curve, if objective test is used, draw bitrate vs. Quality curve;
(5) RD gain or loss can be calculated by the collected four pairs of points.
There are some publications talking about AVC vs. HEVC vs.VP9 as well. I’ll write a more detailed comparison process when I have a chance to do it. FFMPEG, x264 or x265, SSIMplus are the useful tools for this task.
By the way, all operations of my comparisons are executed by Perl script. Do you think a web-based dedicated video codec comparison tool would be interested to the community and has certain market size?