Autotests

Keywords: autotest

Source code validation is a critical step in any software library, particularly for verifying the portability of code to different processors and platforms. Packaged with liquid are a number of automatic test scripts to validate the correctness of the source code. The test scripts are located under each module's tests directory and take the form of a C source file. The testing framework operates similarly to cppunit and cxxtest , however it is written in C. The generator script scripts/autoscript parses these header files looking for the key " void autotest_ " which corresponds to a specific test. The script generates the header file autotest_include.h which includes all the modules' test headers as well as several organizing structures for keeping track of which tests have passed or failed. The result is an executable file, xautotest , which can be run to validate the functional correctness of liquid on your target platform.

You can run all the tests and generate an output report simply with make check . As of mid 2022 there are over 1,000 test scripts and nearly 370,000 checks which cover more than 78% of all of liquid .

Macros

Each module contains a number of autotest scripts which use pre-processor macros for asserting the functional correctness of the source code.

  • CONTEND_EQUALITY\((x,y)\) asserts that \(x==y\) and fails if false.
  • CONTEND_INEQUALITY\((x,y)\) asserts that \(x\) differs from \(y\) .
  • CONTEND_GREATER_THAN\((x,y)\) asserts that \(x \gt y\) .
  • CONTEND_LESS_THAN\((x,y)\) asserts that \(x \lt y\) .
  • CONTEND_DELTA\((x,y,\Delta)\) asserts that \(|x-y| \lt \Delta\)
  • CONTEND_EXPRESSION\((expr)\) asserts that some expression is true.
  • CONTEND_SAME_DATA\((ptrA,ptrB,n)\) asserts that each of \(n\) byte values in the arrays referenced by \(ptrA\) and \(ptrB\) are equal.
  • AUTOTEST_PASS\(()\) passes unconditionally.
  • AUTOTEST_FAIL\((string)\) prints \(string\) and fails unconditionally.
  • AUTOTEST_WARN\((string)\) simply prints a warning.

The autotest program will keep track of which tests elicit warnings and add them to the list of unstable tests.

Here are some examples:

  • CONTEND_EQUALITY (1,1) will pass
  • CONTEND_EQUALITY (1,2) will fail

Running the autotests

The result is an executable file named xautotest which has several options for running. These options may be viewed with either the -h or -u flags (for help/usage information).

$ ./xautotest -h
Usage: xautotest [OPTION]
Execute autotest scripts for liquid-dsp library.
  -h            display this help and exit
  -t <id>       run specific test
  -p <id>       run specific package
  -r            run all tests, random order
  -R <seed>     specify random seed value
  -L            lists all scripts
  -l            lists all packages
  -x            stop on fail
  -s <string>   run all tests matching search string
  -v            verbose
  -q            quiet
  -o <filename> output file (json)

Simply running the program without any arguments executes all the tests and displays the results to the screen. This is the default response of the target make check .

Random Seed

Tests are run by setting the random seed to time(NULL) by default. This allows greater coverage for testing by effectively choosing a different random seed each time the test suite is run; however because repeatability is important for debugging, the random seed can be specified with the -R flag.

Output Report

When the autotest script is run, it generates a JSON file which tabulates the entirety of the test results. This file can be useful for reproducing the results (particularly failures) in specific environments and machine architectures.

{
  "build-info" : {},
  "pass" : true,
  "num_failed" : 0,
  "num_checks" : 369656,
  "num_warnings" : 55,
  "command-line" : " ./xautotest -v -o autotest.json",
  "run-mode" : "RUN_ALL",
  "rseed" : 1660183820,
  "stop-on-fail" : false,
  "tests" : [
    {"id":  0, "pass":true, "num_checks":   1, "num_passed":   1, "name":"null"},
    {"id":  1, "pass":true, "num_checks":   2, "num_passed":   2, "name":"libliquid"},
    {"id":  2, "pass":true, "num_checks":   4, "num_passed":   4, "name":"agc_crcf_dc_gain_control"},
    {"id":  3, "pass":true, "num_checks":   3, "num_passed":   3, "name":"agc_crcf_scale"},
    {"id":  4, "pass":true, "num_checks":   1, "num_passed":   1, "name":"agc_crcf_ac_gain_control"},
    {"id":  5, "pass":true, "num_checks":   1, "num_passed":   1, "name":"agc_crcf_rssi_sinusoid"},
    {"..."},
    {"id":1091, "pass":true, "num_checks":   5, "num_passed":   5, "name":"lshift"},
    {"id":1092, "pass":true, "num_checks":   5, "num_passed":   5, "name":"rshift"},
    {"id":1093, "pass":true, "num_checks":   5, "num_passed":   5, "name":"lcircshift"},
    {"id":1094, "pass":true, "num_checks":   5, "num_passed":   5, "name":"rcircshift"}
  ]
}

Autotest Examples

Run all autotests matching the string " firfilt ":

$ ./xautotest -s firfilt
40: firfilt_xxxf
40: firfilt_xxxf:
    203 :   PASS    passed    8 /    8 checks (100.0%) : firfilt_rrrf_data_h4x8
    204 :   PASS    passed   16 /   16 checks (100.0%) : firfilt_rrrf_data_h7x16
    205 :   PASS    passed   32 /   32 checks (100.0%) : firfilt_rrrf_data_h13x32
    206 :   PASS    passed   64 /   64 checks (100.0%) : firfilt_rrrf_data_h23x64
    207 :   PASS    passed   16 /   16 checks (100.0%) : firfilt_crcf_data_h4x8
    208 :   PASS    passed   32 /   32 checks (100.0%) : firfilt_crcf_data_h7x16
    209 :   PASS    passed   64 /   64 checks (100.0%) : firfilt_crcf_data_h13x32
    210 :   PASS    passed  128 /  128 checks (100.0%) : firfilt_crcf_data_h23x64
    211 :   PASS    passed   16 /   16 checks (100.0%) : firfilt_cccf_data_h4x8
    212 :   PASS    passed   32 /   32 checks (100.0%) : firfilt_cccf_data_h7x16
    213 :   PASS    passed   64 /   64 checks (100.0%) : firfilt_cccf_data_h13x32
    214 :   PASS    passed  128 /  128 checks (100.0%) : firfilt_cccf_data_h23x64

==================================
 PASSED ALL 600 CHECKS
==================================

Run test 405 :

$ ./xautotest -t 405
    405 :   PASS    passed   16 /   16 checks (100.0%) : demodsoft_apsk8
==================================
 PASSED ALL 16 CHECKS
==================================