Skip to main content

VMAF

Under Maintenance

The content in this entry is incomplete & is in the process of being completed.

Short for Video Multimethod Assessment Fusion, VMAF is a full reference video quality assessment algorithm developed primarily by Netflix.

Installation​

Vmaf comes as a part of libvmaf. There are two ways it is commonly used:

  • As an FFmpeg filter
  • As a standalone binary

The instructions below are written for Linux & macOS. On Windows, you can use the Windows Subsystem for Linux to follow along.

In order to build from source, follow the instructions below.

  1. Install the required dependencies via your package manager of choice. The necessary dependencies are nasm, ninja-build, doxygen, & xxd.

  2. Clone the repository & enter the corresponding directory

Clone & Enter
git clone https://github.com/Netflix/vmaf/
cd vmaf/
  1. Compile with meson & ninja
meson setup libvmaf libvmaf/build --buildtype release -Denable_float=true
sudo ninja -vC libvmaf/build install

Now, you can run the VMAF binary with the following command:

vmaf --help

If you would not like to build from source, you may grab the latest build from the VMAF GitHub releases for your operating system.

Now, you can:

/path/to/vmaf --reference refrence.y4m --distorted distorted.y4m

Tip: If the VMAF binary exists but is not market as executable, you might need to chmod +x /path/to/vmaf

Explainer on command line flags can be found here

The disadvantage of using the bin is that you need .yuv|y4m files, you can that overcome by using named pipes Simple example using ffmpeg as a decoder:

# create the pipes
mkfifo ref.pipe
mkfido dist.pipe

# run these each in a new terminal, order docent matter
ffmpeg -v error -i ref.mkv -strict -1 -f yuv4mpegpipe - > ref.pipe
ffmpeg -v error -i dist.mkv -strict -1 -f yuv4mpegpipe - > dist.pipe

# after starting the two ffmpeg processes,
# start the vmaf in a new terminal
/path/to/vmaf --reference ref.pipe --distorted dist.pipe

# delete the pipes after usage
rm ref.pipe dist.pipe

The Advantages of this are:

  • No need for a ffmpeg build with --enable-libvmaf
  • Clear & simple usage of VMAF's various options, like --aom_ctc

Disadvantages are:

  • Difficult/awkward to use without a wrapper script

Scoring​

scores range from 0 to 100, and are best interpreted in a linear way, 100 meaning perfect quality, 0 meaning not recognisable, more info in the Best Practices section here It aligns with mean opinion scores (MOS) really well at low/medium bitrates, as stated in this benchmark

Some weaknesses​

  • Newer codecs like AV1 and VVC introduce new kinds of artifacting that v0.6.2 (current model as of Jan 2024) doesn't recognise, that's why its performance might degrade, for example, high motion scenes being affected badly
  • It's bad at "transparent" levels of quality, kinds of quality that the average viewer might not notice
  • Synthetic grain throws off scores, this issue is not isolated to vmaf, but it should be noted regardless
With ffmpeg you can disable application of synthetic grain

place -filmgrain 0 before -i in the above ffmpeg commands, limited to decoding with dav1d TODO: replace this tip with an export_side_data solution

  • As of January 2024, it doesn't work on HDR content, nothing prevents you from feeding it un-tonemapped PQ but scores will be off
  • In contrast with SSIMULACRA2, it focuses on appeal, not necessarily on fidelity. They often align, but not always.
  • Due to the ML nature, comparing the same video to itself will not always result in a score of 100

Comparing to SSIMULACRA2​

One big advantage over SSIMULACRA2 is the inclusion of some temporal information. This means that VMAF weights frames that have a lot of motion higher. Meanwhile, SSIMULACRA2 based solutions compare each frame to the reference frame individually, since it is an image metric at heart. VMAF also wins in speed and general ease of use.

Additional resources