Skip to main content

AV1 Encoding for Dummies

· 17 min read
Simulping
Simulping
Maintainer / Encoder
Gianni Rosato
Gianni Rosato
Maintainer

This guide will show you how to encode in AV1 the right and optimal way. Yes, you using standalone libaom, libsvtav1, and librav1e from FFmpeg or even piping yuv4mpeg into mainline aomenc are all unoptimal.

Outdated Information

While a lot of the information presented in this guide is still relevant and correct, the AV1 ecosystem has changed dramatically since this guide's inception which has demanded a rewrite. Please see the AV1 for Dummies blog post for more information.

Compare

In this guide, we'll be installing Av1an for chunked encoding and infinite threading, because the current state of AV1 encoders, except for SVT-AV1, unfortunately lacks threading and will only use very low amount of cores, which hampers speeds. The only caveat to this approach is RAM consumption, encoding 2160p (4K) with aomenc with 4 workers could take upwards of 16GB of RAM! So do keep this in mind.

Installing the Tools​

Given all of the different operating systems that people use on a day to day basis and the various different encoding workflows that exist, there are a number of ways to do this.

Jump to: Windows | macOS | Linux

Microsoft Windows​

The GUI Way​

  1. Install NMKODER which is a GUI front-end to av1an with all dependencies installed.
  2. You're done, you can skip to the encoding part
Almost abandonware

Since Nmkoder already ships everything by default and its last release was 29th March 2022. You need to manually update all encoders and tools to get better encoding speeds. Missing out on updates will result in your encodes being sub-optimal.

The WSL2 Way​

(Recommended)

If you're not already familiar with WSL2, the The Windows Subsystem for Linux (WSL) is a feature of the Windows operating system that allows you to run a Linux file system, along with Linux command-line tools and GUI apps, directly on Windows. This lets Linux distributions run on bare metal without managing any virtual machines, so encoding performance is very good.

The easiest way to encode with WSL2 is to use rAV1ator CLI, an interactive TUI for Av1an. An ArchWSL2 installation tutorial is provided here.

The Automated Way​

There is now a batch script for automating the install process, which can be found here. The instructions are in the README file.

caution

The script will download outdated version encoders and tools such as aom-av1-psy and MKVToolNix v76.0, if you are fine with these you can proceed.

The Manual Way​

  1. Install Python 3.10.x, this will change so consult from the Vapoursynth website if you're reading this from the future from here and select "Windows Installer 64-bit". Upon installation check the tick for adding Python to PATH like so Python PATH)

  2. Download and install Vapoursynth from here and select "VapourSynth64-RXX.exe"

  3. Open the terminal and type vsrepo.py install lsmas ffms2 to install some plugins for Av1an to work.

  4. Download MKVToolNix from here, select "mkvtoolnix-64bit-XX.X.X-setup.exe", and install (Also available on winget!)

  5. Download Av1an from here (SELECT LATEST AND CLICK THE "ASSETS" DROPDOWN)

  6. Download shared libraries FFmpeg from gyan.dev

  7. Download a pre-built fork of Aomenc (aom-av1-lavish) which has neat stuff such as sane defaults, new tunes, optimizations, etc. This can be downloaded for Windows here (Current as of Sept 6, 2023)

info

If you opt to compile aomenc yourself, you can view the instructions on how to do that here.

  1. Move Av1an, FFmpeg (Including the FFmpeg DLLs), and aomenc to somewhere preferable, eg C:\Encoding.
  2. Add the folder AND MKVTOOLNIX INSTALLATION FOLDER to the Windows PATH environment.

macOS​

macOS is very similar to Linux, although there aren't any GUI tools for AV1 encoding that I can comfortably recommend.

Homebrew + Macports for Av1an + rav1e: Note that some commands may have to be run with sudo, which I won't explicitly include for security reasons.

Installing the Homebrew package manager is a well documented process at this point:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

As is installing MacPorts. Install the relevent .pkg for your macOS version from the MacPorts Project website: www.macports.org/install.php

Now, you can run the following commands:

brew update && brew upgrade
brew install rav1e aom mkvtoolnix ffmpeg
# Usually you must run MacPorts commands for package installations as root
port upgrade outdated
port install av1an

This is the easiest way to get everything set up & working to produce AV1 video with rav1e or mainline aomenc & Av1an. You can check that things are installed by running the following commands & parsing their output:

% av1an --version
av1an 0.4.1-unstable (rev e10880d) (Release)

* Compiler
  rustc 1.70.0 (LLVM 16.0)

* Target Triple
  aarch64-apple-darwin

* Date Info
  Commit Date:  2023-06-25

* VapourSynth Plugins
  systems.innocent.lsmas : Not found
  com.vapoursynth.ffms2  : Not found
% rav1e --version | grep "release" -C 1
rav1e 0.6.6 () (release)
rustc 1.69.0 (84c898d65 2023-04-16) (built from a source tarball) aarch64-apple-darwin
% aomenc --help | grep "AOMedia" -C 3

Included encoders:

    av1    - AOMedia Project AV1 Encoder 3.6.1 (default)

        Use --codec to switch to a non-default encoder.

Notice systems.innocent.lsmas : Not found in the Av1an output. This means you won't be able to use the lsmash chunking method through vapoursynth & may instead have to rely on hybrid chunking, through -m hybrid. This is slower & takes up disk space while encoding, but still works. A sample Av1an command with this basic installation may look like this:

av1an -i "input" -y --resume --verbose --split-method av-scenechange -m hybrid -c mkvmerge -e rav1e --force -v " --tiles 8 -s 4 --quantizer 80 --no-scene-detection" --photon-noise 7 --chroma-noise --pix-format yuv420p10le -w 8 -o "output.mkv"

Building From Source

If you want lsmash support, aom-av1-lavish instead of mainline, or anything else that isn't covered by the more basic installation, you'll have to compile from source. Things are very similar to Linux, with a few oddities:

  • macOS sometimes doesn't have a /usr/local/bin by default. You can fix this by doing mkdir /usr/local/bin.
  • Homebrew installs everything in its own directory structure. If you're building things from source that rely on libraries from vapoursynth, zimg, lsmash, etc, make sure to copy them from /opt/homebrew/lib to /usr/local/lib. Finding them is a matter of ls | grep "keyword" & copying what looks reasonable to be associated with the tool you're using.
  • Building most things from source will have instructions for *nix which work for both macOS & Linux. Even if it says Linux, there's a good chance it'll work on macOS as well, & it is always worth trying Linux build instructions on Mac. I won't be going through building every encoding tool & dependency from source, as it is generally much more intuitive than Windows, but building Av1an is worth detailing here just as an example.
brew install git rust nasm
git clone https://github.com/master-of-zen/Av1an
cd Av1an
RUSTFLAGS="-C target-cpu=native" cargo build --release
cd .. && cd target/release
cp av1an /usr/local/bin

More Difficult: Building aom-av1-lavish from Source

If you want to make the most out of your hardware & eke out every last drop of quality, it may be worth building aom-av1-lavish from source. The first step is to clone it from the Endless Merging branch:

git clone https://github.com/Clybius/aom-av1-lavish -b Endless_Merging
cd aom-av1-lavish

Now, you need to make some manual changes to the source code until Clybius merges this commit.

  • Add the line #include "aq_variance.h" at line 19 in av1/encoder/encodeframe_utils.c
  • Comment out line 2546 in av1/encoder/speed_features.c. This line is const int qindex_thresh_cdef_sf_s1_s3_l2[2] = { 92, 48 }; & becomes // const int qindex_thresh_cdef_sf_s1_s3_l2[2] = { 92, 48 };.

Now you can continue to build according to the Linux instructions below. Obviously you'll need cmake, which you can install with homebrew along with any other tools you may need. While still in the aom-av1-lavish directory:

mkdir -p aom_build && cd aom_build
cmake .. -DBUILD_SHARED_LIBS=0 -DENABLE_DOCS=0 -DCONFIG_TUNE_BUTTERAUGLI=0 -DCONFIG_TUNE_VMAF=0 -DCONFIG_AV1_DECODER=0 -DENABLE_TESTS=0 -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS="-flto -O3 -march=native" -DCMAKE_C_FLAGS="-flto -O3 -march=native -pipe -fno-plt" -DCMAKE_LD_FLAGS="-flto -O3 -march=native"
make -j$(nproc)
# This may need to be run as root:
make install

Now you can run aomenc --help | grep "AOMedia" -C 3 to see if lavish installed. If you're getting the same output as above, you may need to copy the aomenc executable to /opt/local/bin, /usr/local/bin, & /opt/homebrew/bin if you already installed mainline aomenc. Running the version info command again, the correct output should look something like this:

% aomenc --help | grep AOMedia -C 3

Included encoders:

    av1    - AOMedia Project AV1 Encoder Psy v3.6.0 (default)

        Use --codec to switch to a non-default encoder.

Notice how it says AOMedia Project AV1 Encoder Psy instead of AOMedia Project AV1 Encoder. You should be all set after this to start using aom-av1-lavish & following the current parameter meta as outlined below.

Linux​

info

Yet again, try using Arch. It's way easier.

The GUI Way​

  • Install Aviator (SVT-AV1 + FFmpeg) or rAV1ator basically same thing but Av1an + rav1e. Both are only available as Flatpaks. Keep in mind Aviator ships with SVT-AV1 and rAV1ator with rav1e instead of aomenc/AOM-AV1, which I will not be covering here.

The TUI Way​

(Recommended)

  • Install rav1ator-cli, a TUI for using Av1an meant to be easy to use. Much more flexible than the GUI options & can work with a number of encoders. See this page for more info. Can be easily used on any distro.

The Compiling Route​

Ubuntu​

The guide below is targeted towards 22.04, packages and other things may be different on other versions. First Install Rust via rustup first, as apt version of Rust is severely outdated, then you can continue.

Install dependencies:

sudo apt install wget python unzip unrar build-essential meson autoconf automake libtool git nasm yasm python3-dev python3-pip cython3 libass-dev libqt5websockets5-dev libfftw3-dev libtesseract-dev ffmpeg libavcodec-dev libavformat-dev libswscale-dev libavutil-dev libswresample-dev libmediainfo-dev mkvtoolnix mediainfo perl nasm yasm git cmake libavutil-dev libavcodec-dev libavformat-dev libavdevice-dev libavfilter-dev libswscale-dev libswresample-dev libpostproc-dev llvm libclang-dev libssl-dev

Install l-smash:

git clone https://github.com/l-smash/l-smash.git
cd l-smash
./configure --enable-shared --extra-cflags="-march=native"
make -j$(nproc)
sudo make install

Install zimg:

git clone --recursive https://github.com/sekrit-twc/zimg.git
cd zimg
./autogen.sh
./configure
make -j$(nproc)
sudo make install

Install ImageMagick:

git clone https://github.com/ImageMagick/ImageMagick
cd ImageMagick
./configure
make -j$(nproc)
sudo make install

Install Vapoursynth R63:

wget https://github.com/vapoursynth/vapoursynth/archive/refs/tags/R63.zip
unzip R63.zip
cd vapoursynth-R63
./autogen.sh
./configure CFLAGS="-march=native" CXXFLAGS="-march=native" --libdir=/usr/lib
make -j$(nproc)
sudo make install
sudo mkdir /usr/lib/vapoursynth
sudo ldconfig

The plugin directory will be located in /usr/lib/vapoursynth.

Install L-SMASH-Works Vapoursynth Plugin:

git clone https://github.com/AkarinVS/L-SMASH-Works -b ffmpeg-4.5
cd L-SMASH-Works/VapourSynth && mkdir build && cd build
meson .. --optimization=3 --default-library=static -Db_lto=true -Dc_args="-march=native" -Dcpp_args="-march=native"
ninja -j$(nproc)
sudo cp libvslsmashsource.so /usr/lib/vapoursynth/
danger

L-SMASH-Works doesn't work on aarch64, it is recommended to use other plugins instead.

Install FFMS2 Vapoursynth Plugin:

git clone https://github.com/FFMS/ffms2
cd ffms2
./autogen.sh
./configure CFLAGS="-O3 -march=native" CXXFLAGS="-O3 -march=native"
make -j$(nproc)
sudo cp src/core/.libs/libffms2.so src/core/.libs/libffms2.so.5 src/core/.libs/libffms2.so.5.0.0 /usr/lib/vapoursynth

Install Av1an:

git clone https://github.com/master-of-zen/Av1an
cd Av1an
RUSTFLAGS="-C target-cpu=native" cargo build --release
sudo cp target/release/av1an /usr/local/bin

When there's no errors, proceed to compiling aom-av1-lavish.

Arch​

Install dependencies:

sudo pacman -S vapoursynth ffmpeg av1an mkvtoolnix-gui git perl cmake ninja meson nasm vapoursynth-plugin-lsmashsource ffms2

you're done, proceed.

Compiling aom-av1-lavish​

git clone https://github.com/Clybius/aom-av1-lavish -b Endless_Merging
cd aom-av1-lavish && mkdir -p aom_build && cd aom_build
cmake .. -DBUILD_SHARED_LIBS=0 -DENABLE_DOCS=0 -DCONFIG_TUNE_BUTTERAUGLI=0 -DCONFIG_TUNE_VMAF=0 -DCONFIG_AV1_DECODER=0 -DENABLE_TESTS=0 -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS="-flto -O3 -march=native" -DCMAKE_C_FLAGS="-flto -O3 -march=native -pipe -fno-plt"
make -j$(nproc)
sudo make install

Encoding​

The moment you've all been waiting for, let's just get into it. Here's an example recommended parameter as of now (09/03/23) [MM/DD/YY]:

av1an -x 300 -i input.mkv -w 4 -e aom -c mkvmerge --resume -m lsmash --photon-noise=10 --set-thread-affinity=2 --verbose -a " -an " -f " -an " -v " --bit-depth=10 --cpu-used=4 --end-usage=q --cq-level=24 --threads=2 --tile-columns=0 --tile-rows=0 --lag-in-frames=64 --tune-content=psy --tune=ssim --enable-keyframe-filtering=1 --disable-kf --kf-max-dist=9999 --enable-qm=1 --deltaq-mode=0 --aq-mode=0 --quant-b-adapt=1 --enable-fwd-kf=0 --arnr-strength=1 --sb-size=dynamic --enable-dnl-denoising=0 " -o "output.mkv"
Parameter Meta

It is strongly recommended to join the AV1 Discord server to get the latest updates on what to use and which to set, as it's the only easily reachable place for everything AV1 & encoding tips in general.

Now let's dissect it one-by-one

Av1an parameters:

  • -i Input.

  • -x 300 Sets scene split length to 300 frames, you can increase it for more quality at the tradeoff of video seekability.

  • -w 4 Specifies the amount of "workers" or amount of encoders working on the video.

  • --verbose Sets logging to verbose.

  • --resume Resumes the encode even when you haven't encoded yet. I strongly recommend leaving this if you resume a lot since you can accidentally delete your whole progress (There's no delete confirmation feature.. yet) if you "resumed" without the parameter in place.

  • -e aom Specifies we're using aomenc encoder which should be the default option.

  • -c mkvmerge Specifies we're using mkvmerge (MKVToolNix) to concatenate the parts when done, you can specify with ffmpeg if you want to but this is the best method.

  • -m lsmash Specifies we're using l-smash (Vapoursynth plugin) to split the videos, this is also the best method because ffms2 causes video lag (Tested a year ago, might change now) and other methods just suck (Slow and not worth it, learned the hard way). You can attempt to use ffms2 when inputting VC-1 videos as it is not possible with l-smash (Or convert it to lossless with x264 qp 0).

  • -f " -an " -f Stands for ffmpeg parameters, -an is to remove all audio since its better to encode and merge it separately. To crop use -f " -an -vf crop=1920:800 " for example to crop the video to 1920x800.

  • -v " " Is where you put the encoder's parameters in.

  • -a " -an " FFmpeg audio encoding options, we're removing it cause we can always add it later. But if you want to, you can also encode directly. Here's an example for encoding to Opus using libopus assuming stereo: -a " -c:a libopus -b:a 128k ".

  • --photon-noise=10 AV1 grain synthesis, which is a technique where the encoder puts fake grain in so it looks more natural and potentially hiding video artifacts (cause grain is hard to encode and explodes bitrate usage because of their randomness), 5-8 for almost none to little grain, 10-14 for medium, 15+ heavy, 20+ extremely heavy, 30+ for extremely grainy 90s live action films.

  • --set-thread-affinity=2 Pins the thread to the encoder, aligns with --threads=2 in the encoder parameter so set them accordingly.

aomenc parameters:

  • --bit-depth=10 We're using 10bit because it makes the video smaller and reduces banding.

  • --cpu-used=4 This is the preset which ranges from 0-9, you can go to 3 if you want more efficiency, 2 if you have a lot of time, 4 is the sweet spot, and 6 if you want speed. Don't go above 6 (Worst efficiency) or even 0 (It would take WEEKS to finish).

  • --end-usage=q --cq-level=24 This specifies that we are going to use a knockoff version of CRF level similar to x264/x265 encoders, in this case CRF 24.

  • --threads=2 Sets the amount of threads the encoder can use, aligns with --set-thread-affinity in Av1an.

  • --tile-columns=0 --tile-rows=0 This is the tiles options, where the encoder splits the videos into tiles to encode faster, see the image below (Yellow lines):

Tiling
Tile usage

Do NOT use tiles for 1080p and below, use 1 tile-columns at 1440p (2K), 2 tile-columns and 1 tile-rows for 2160p (4K)

  • --lag-in-frames=64 Similar to x264/x265 rc-lookahead. Sets a number of frames to look ahead for frametype and ratecontrol, allowing for better compression decision making. Setting to a value greater than 64 is generally not considered useful.

  • --aq-mode adaptive quantization mode, 0 is better most of the time

  • --tune-content=psy --tune=ssim As the name suggests they are tunes that affect the video output, for the better, and for the worst

Tunes to use

Set tune-content to animation if you're encoding above cq-level=30 A.K.A lower quality, despite it's name Set tune-content to psy for everything else, do not use if you encode above cq-level=30 For tune, this is a bit tricky. For now, the meta seems to be ssim, but back then it was lavish which is considered THE best tune because it's based on butteraugli. Now it's fallen behind because its more blurry than ssim, and before that it was butteraugli, and then ipq_vmaf_psy, and finally just ipq. If you use any of the VMAF tunes, you need to specify --vmaf-model-path= to where you put it.

  • --enable-keyframe-filtering=1 We're setting it to 1 because of compatibility reasons, 2 is more efficient but there are seeking issues and FFmpeg for some reason can't input it.

  • --sb-size=dynamic Allows the encoder to use 128x128 block partitioning besides 64x64 which gives an efficiency boost, ignore it.

  • --deltaq-mode set to 0 because its just better.

  • --arnr-strength=1 Controls how strong the filtering will be, 1 is good for 3D Pixar CGI-like and 2D animation, use 4 if you're doing live action content. Using maximum at higher bitrates would just result in a blurry mess.

  • --disable-kf --enable-fwd-kf=0 We're disabling keyframes cause Av1an already did scene detection, so we wont have to.. And it speeds things up.

  • --kf-max-dist=9999 Maximum keyframe interval, we're setting it at the highest possible value since av1an's scene detection keyframe interval is already 240 by default

  • --enable-chroma-deltaq=1 --enable-qm=1 --quant-b-adapt=1 Parameters that give you free efficiency boost.

  • --enable-dnl-denoising=0 Disables the encoder's built-in denoising technique when grain synthesis is enabled, you can optionally set it to 1 when you have a pretty noisy video since it works quite well.

Concatenation Error on Linux

Run ulimit -n 200000, resume, and it should concatenate just fine. If it still errors, head to the encode directory > encode, and run mkvmerge @../options.json

Merging Everything​

Once you're done just encode your audio using ffmpeg (or just passthrough it), subtitles should be carried along with your video output, and merge them in MKVToolNix! Don't want Matroska files? That's fine, you can use FFmpeg or MP4Box to output into mp4, just keep in mind that PGS/SUP/VOBSUB subtitles are not supported and Opus audio support is still experimental.

Tips & Tricks​

  • --denoise-noise-level=10 Alternative to photon-noise, slower than photon-noise and is the OG grain synthesis method, performs okay and just serves as an alternative. Don't attempt to use it at high values (>12) since it creates noticeable grain patterns.

  • --arnr-maxframes to set max reference frames that will be used to filter the encode, higher values would make the video blurrier at high fidelity but look better at lower bitrates.

  • --butteraugli-resize-factor=2 if you use any of the butteraugli-based tunes (lavish, butteraugli) to speed it up without much losses and --butteraugli-intensity-target=250 to match the content light level.

Final Thoughts​

Encoding has always been about experimentation for the best, there is really no "One size fits all" for encoding content, as they differ from scene complexity, how it's captured (2D/Real life), film grain, dark scenes, etc. So experiment away for your specific type of content!

Guide originally hosted on https://rentry.co/AV1, rewrite and migration by Simulping.