Skip to main content

Av1an

Av1an with 96 workers

Av1an is a video encoding framework. It can increase your encoding speed and improve CPU utilization by running multiple encoder processes in parallel. Target quality, VMAF plotting, and more, available to take advantage for video encoding.

The only downside to this, is that RAM usage increases significantly with each "worker" you add, so if you have a low amount of RAM (< 8GB), this might not be that useful to you.

caution

Av1an does not ship with any dependencies except for it's Docker image, so you will have to provide them on your own.

Prerequisites​

You will need Python, FFmpeg and Vapoursynth to be installed regardless of your current operating system.

Installation​

Windows​

Script​

Windows users can use this script which installs everything needed in a single portable folder. Do note that you will need to update the dependencies manually once installed.

Pre-built Binary​

Av1an is available as a pre-built binary under the "latest" tag on GitHub releases, you can simply download that and place it wherever you want.

Compile from Source​

To compile from source, it is easier to use mingw-w64 which comes with MSYS2. Once installed, open MinGW64 and run the following:

pacman -Syuu; pacman -S cmake git nasm python3 mingw-w64-x86_64-gcc mingw-w64-x86_64-cmake base-devel mingw-w64-x86_64-ffmpeg mingw-w64-x86_64-rust mingw-w64-x86_64-lld mingw-w64-x86_64-clang mingw-w64-x86_64-make

Then, before you do anything further. Download Vapoursynth portable (VapourSynth64-Portable-RXX.7z) from its GitHub release page. Make sure the version you chose is compatible with the current MinGW64 Python version. For example, R65 supports 3.8 and 3.11, the version from Pacman (currently) is 3.11, so you should choose that.

After downloading, open the 7z file and head to \sdk\lib64, copy all the libraries from there to MinGW64's lib directory, this usually resides in C:\msys64\mingw64\lib.

Now finally, resume your session and run the following:

git clone https://github.com/master-of-zen/Av1an
cd Av1an
RUSTFLAGS="-C target-cpu=native" cargo build --release

The binary will be available in C:\msys64\home\USER\Av1an\target\release. The USER part is your Windows username.

macOS​

Using MacPorts​

Av1an is available in the MacPorts repository, to install just simply run:

port install av1an

Compile from Source​

You need to have Git, Nasm, and Rust to be installed first. You can achieve this by using Homebrew or similar.

git clone https://github.com/master-of-zen/Av1an
cd Av1an
RUSTFLAGS="-C target-cpu=native" cargo build --release

Binary is located in Av1an/target/release, you can optionally place it somewhere like /usr/local/bin.

Linux​

Package Managers​

Av1an is available in the package manager of some distributions such as Arch Linux. If not, you can skip to the next part every Linux user is familiar with.

Compile from Source​

Just like macOS, you need to have Git, Nasm, and Rust to be installed. Can be achieved by either using your package manager or Rustup for Rust.

git clone https://github.com/master-of-zen/Av1an.git
cd Av1an
RUSTFLAGS="-C target-cpu=native" cargo build --release

Binary is located in Av1an/target/release, you can optionally place it somewhere like /usr/local/bin or /usr/bin.

Docker​

The following examples assume the file you want to encode is in your current working directory.

docker run --privileged -v "$(pwd):/videos" --user $(id -u):$(id -g) -it --rm masterofzen/av1an:latest -i S01E01.mkv {options}
info

To specify a different directory to use you would replace $(pwd) with the directory

docker run --privileged -v "/c/Users/masterofzen/Videos":/videos --user $(id -u):$(id -g) -it --rm masterofzen/av1an:latest -i S01E01.mkv {options}

The --user flag is required on Linux to avoid permission issues with the docker container not being able to write to the location, if you get permission issues ensure your user has access to the folder that you are using to encode.

caution

The Docker image ships with the default dependencies.

Installing Dependencies​

You will need a chunk method installed as the built-in ones are very slow.

Troubleshooting​

"Error: The file 'XXXXX.ivf' could not be opened for reading: open file error." with mkvmerge on Linux​

Since mkvmerge opens all the encoded chunks at once, this obviously hits the default open file descriptor limit of 1024 on longer videos. You can temporarily increase it with ulimit -n 20000 to allow opening 20,000 files at once in that session.

Gray screen flashing for a single frame in output​

This is a unique problem with Av1an. Since Av1an does random seeking for chunks, traditional keyframe-based decoding methods such as L-SMASH and FFMS2 may fail and result in a gray frame. There are currently only two known methods to solve this:

  1. Create a lossless intermediary with x264 -qp 0, this always fixes all related seeking issues but will result in a huge file due to it's lossless nature.
  2. Using a linear-decoding chunk method such as bestsource, the downside to this is it's mega slow. Tests shown encode speeds drop by a whopping 10-15x slower.
  3. Using dgdecnv chunk method. dgdecnv and DGIndexNV is a proprietary CUVID-accelerated Vapoursynth "Source" decoder created by Donald Alan Graft where DGIndexNV is the standalone program for seeking and dgdecnv is the Vapoursynth plugin itself. It is especially made with random seeking in mind which makes it perfect for Av1an. Only downside is you need an NVIDIA GPU and DGIndexNV only supports x86_64 platforms.