Review: Moscow State University Video Quality Measurement Tool
I’ve been compressing video files since 1991, and I’ve never worked with a more useful analysis tool than the Moscow State University Video Quality Measurement Tool (VQMT). Though it’s Windows-only and, at $999, not inexpensive, it’s worth it for any streaming producer making serious encoding-related decisions.
Here are the appealing features: first, VQMT can calculate 19 different objective quality metrics, such as peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and video quality metric (VQM), which quantify the quality of your encoded files. Second, it provides a great interface for exploring the subjective quality of your encoded files. Finally, there’s a batch interface you can use to produce objective metrics when you’ve got a bunch of files to test.
The Problem
Let’s add some context to this analysis and use VQMT to help resolve a real concern. One major issue facing streaming producers is the need to produce multiple groups of streams for different targets. That is, while computers can play files encoded using H.264’s High profile, many older iOS devices can’t, and Google still recommends using the Baseline profile for all encodes targeting Android devices. So many producers create one set of streams using the High profile for computers, and another using the Baseline profile for mobile devices. This adds encoding and storage costs and can decrease the overall cacheability of the files.
I recommend encoding a single set of files using the Baseline profile, because in my tests, the subjective quality differences between files encoded using the Baseline and High profiles weren’t that significant (for more information, see “H.264 in a Mobile World: Adios to the Main and High Profiles”). Let’s use VQMT to objectify and refine the subjective comparisons.
To do so, I encoded a 640x360 input file to 600Kbps and 1200Kbps at the same resolution in Sorenson Squeeze using the x264 codec (Very Slow preset), with constant bitrate encoding and a keyframe interval of 3 seconds. I encoded each file twice, once using the Baseline profile, once using the High profile, and verified that file sizes were within 1 percent. Let’s see how they differed in quality objectively and subjectively.
The VQMT Workflow
Using VQMT is relatively simple, but there are a couple of caveats. First, when you compare encoded files, they must be at the same resolution as the source file. That’s because objective quality metrics such as PSNR and VQM work by comparing the encoded file to the original and quantifying the differences. If your source file is 1080p and your output file 360p, the differences are largely meaningless. That’s not a huge deal -- it just means that you have to scale your source files down to your target output resolution before encoding.
In addition, VQMT can only work with files in AVI, AVS, YUV and BMP formats; you can’t directly analyse MP4 files or MOV files. In my work, I’ve standardised on the YUV format and use FFmpeg, a free cross-platform tool you can download at ffmpeg.org/download.html, to convert my files to YUV. FFmpeg is amazingly flexible, and I’ve used the tool to convert a range for files to YUV, including MOV, MP4, HEVC, and WEBM files. If you don’t know FFmpeg, there’s lots of help on the internet, or you can pick up Frantisek Korbels’ book FFmpeg Basics on Amazon. Or you can use a GUI-based tool, though FFmpeg’s batch operation is tough to beat.
Once you’ve got your files converted, it’s time to load them into the program. VQMT is shown in Figure 1, with all the steps labelled separately in the program interface. For Step 1, you load the source file, then the compressed files, clicking the Comparative analysis check box when comparing two files, as we are here. Then you load the two compressed files. For Step 2, you choose a test in the Metric Selection drop down list, which in the figure is set to VQM.
Figure 1. VQMT’s main interface
Next, you choose the Output Selection (Step 3), which can be a CSV spreadsheet file containing all the comparisons and a computed average score, a metric visualisation, and/or “bad frames,” or a selection of frames showing the best and worst of the encoded files. In my use, I enabled the CSV file, but didn’t use metric visualisations or bad frames, preferring to use the techniques discussed below to subjectively compare the files.
Related Articles
Any good compressionist needs tools to help reveal critical data about encoded files, from codecs and resolution to GOP structure and quality levels. Here's a look at some of the most useful.
18 Mar 2020
Video quality measurement is more crucial than ever. In this buyers' guide, we'll identify the most commonly used metrics and discuss tools that deploy them.
20 Mar 2019
Reach viewers no matter where they are. Check out these strategies for reaching the most devices with the smallest number of files.
06 Mar 2013
Get advice from Sorenson, Zencoder, and Encoding.com: When companies should encode and transcode their own content, and when it makes sense to send video to the cloud.
28 Sep 2012
Our annual test of the leading desktop encoding programs.
10 Mar 2011