FFmpeg: Create FATE tests

peter_b · Post by **peter_b** » Sun Aug 30, 2015 6:05 pm

I'm going to write down some notes while trying to create new tests for FFmpeg's Automated Test Environment (FATE).

Prepare the test environment:
- Clone and checkout current FFmpeg git sourcecode, as describe on FFmpeg's Download Page:

Code: Select all

$ git clone git://source.ffmpeg.org/ffmpeg.git ffmpeg

Install the packages required to build your desired FFmpeg flavor, as you would normally.

Download the FATE suite samples:
FATE requires a set of samples as input to run properly. These are called "fate-suite".
You can tell the build environment where you want to put these samples.
For example:

Code: Select all

$ make fate-rsync SAMPLES=../fate-suite/

This puts the samples outside the FFmpeg source tree, in the "../fate-suite/" folder.
The rsync download can take a while. At the time of this writing, the fate-suite had 1.3 GB size.

When adding your own FATE tests, you will want to re-run the tests several times, without telling it every time where to find the samples.
Therefore, you can tell the build environment during "./configure" where to look for them:

Code: Select all

$ ./configure --samples=../fate-suite/

Compile/build:
Now, compile FFmpeg and run the tests before adding your own, to see if everything's alright:

Code: Select all

$ make fate

Add a test:
First, read the documentation articles about how to add a new FATE test.
Since running all FATE tests takes a long time, you might want to run just your specific test(s).
Use "make fate-list" to see which make-targets for FATE tests exists.

For example, to see all FATE tests for FFV1 codec, run:

Code: Select all

$ make fate-list | grep ffv1

So, to run only the fate-vsynth1 tests for FFV1.3, run:

Code: Select all

$ make fate-vsynth1-ffv1.3

In this HowTo, I'll refer to your own test targets as "fate-mytest-target". Replace that target name with yours accordingly.

After you add a test, "make fate" will throw an error, because the reference output for it will be missing (of course). This might look something like this:

TEST vsynth2-ffv1.3
reference file './tests/ref/vsynth/vsynth2-ffv1.3' not found
./tests/fate-run.sh: 282: ./tests/fate-run.sh: cannot open tests/data/fate/vsynth2-ffv1.3.diff: No such file
Test vsynth2-ffv1.3 failed. Look at tests/data/fate/vsynth2-ffv1.3.err for details.
make: *** [fate-vsynth2-ffv1.3] Error 1

Run it again, but tell make to generate the reference using "GEN=1":

Code: Select all

$ make GEN=1 fate-mytest-target

You'll now see that MD5 hashcodes are calculated for the generated output files.
Also not the line starting with "GEN":

TEST vsynth2-ffv1.3
--- ./tests/ref/vsynth/vsynth2-ffv1.3 2015-08-30 17:56:27.111424613 +0200
+++ tests/data/fate/vsynth2-ffv1.3 2015-08-30 17:58:28.196005945 +0200
@@ -0,0 +1,4 @@
+6d7b6352f49e21153bb891df411e60ec *tests/data/fate/vsynth2-ffv1.3.avi
+3718026 tests/data/fate/vsynth2-ffv1.3.avi
+36d7ca943916e1743cefa609eba0205c *tests/data/fate/vsynth2-ffv1.3.out.rawvideo
+stddev: 0.00 PSNR:999.99 MAXDIFF: 0 bytes: 7603200/ 7603200
GEN ./tests/ref/vsynth/vsynth2-ffv1.3

Now the reference files should have been created. Remove "GEN=1" and run the tests again.
The result should now be just the lines starting with "TEST". Like this:

TEST vsynth2-ffv1.3

Test verbosity:
To see what code is executed for your FATE tests, use the shell variable "V" to set the verbosity level:

Verbosity level, can be set to 0, 1 or 2.
0: show just the test arguments
1: show just the command used in the test
2: show everything

Test threading:
According to FFmpeg's FATE user documentation, there are 2 shell variables to define multi-threading behavior:
THREADS: Specify how many threads to use while running regression tests, it is quite useful to detect thread-related regressions.
THREAD_TYPE: Specify which threading strategy test, either ‘slice’ or ‘frame’, by default ‘slice+frame’
I think, the default is THREADS=1.

Links: