Autotests
Keywords: autotest
Source code validation is a critical step in any software library, particularly for verifying the portability of code to different processors and platforms. Packaged with liquid are a number of automatic test scripts to validate the correctness of the source code. The test scripts are located under each module's tests directory and take the form of a C source file. The testing framework operates similarly to cppunit and cxxtest , however it is written in C. The generator script scripts/autoscript parses these header files looking for the key " void autotest_ " which corresponds to a specific test. The script generates the header file autotest_include.h which includes all the modules' test headers as well as several organizing structures for keeping track of which tests have passed or failed. The result is an executable file, xautotest , which can be run to validate the functional correctness of liquid on your target platform.
You can run all the tests and generate an output report simply with make check . As of mid 2022 there are over 1,000 test scripts and nearly 370,000 checks which cover more than 78% of all of liquid .
Macros∞
Each module contains a number of autotest scripts which use pre-processor macros for asserting the functional correctness of the source code.
- CONTEND_EQUALITY\((x,y)\) asserts that \(x==y\) and fails if false.
- CONTEND_INEQUALITY\((x,y)\) asserts that \(x\) differs from \(y\) .
- CONTEND_GREATER_THAN\((x,y)\) asserts that \(x \gt y\) .
- CONTEND_LESS_THAN\((x,y)\) asserts that \(x \lt y\) .
- CONTEND_DELTA\((x,y,\Delta)\) asserts that \(|x-y| \lt \Delta\)
- CONTEND_EXPRESSION\((expr)\) asserts that some expression is true.
- CONTEND_SAME_DATA\((ptrA,ptrB,n)\) asserts that each of \(n\) byte values in the arrays referenced by \(ptrA\) and \(ptrB\) are equal.
- AUTOTEST_PASS\(()\) passes unconditionally.
- AUTOTEST_FAIL\((string)\) prints \(string\) and fails unconditionally.
- AUTOTEST_WARN\((string)\) simply prints a warning.
The autotest program will keep track of which tests elicit warnings and add them to the list of unstable tests.
Here are some examples:
- CONTEND_EQUALITY (1,1) will pass
- CONTEND_EQUALITY (1,2) will fail
Running the autotests∞
The result is an executable file named xautotest which has several options for running. These options may be viewed with either the -h or -u flags (for help/usage information).
$ ./xautotest -h
Usage: xautotest [OPTION]
Execute autotest scripts for liquid-dsp library.
-h display this help and exit
-t <id> run specific test
-p <id> run specific package
-r run all tests, random order
-R <seed> specify random seed value
-L lists all scripts
-l lists all packages
-x stop on fail
-s <string> run all tests matching search string
-v verbose
-q quiet
-o <filename> output file (json)
Simply running the program without any arguments executes all the tests and displays the results to the screen. This is the default response of the target make check .
Random Seed∞
Tests are run by setting the random seed to time(NULL) by default. This allows greater coverage for testing by effectively choosing a different random seed each time the test suite is run; however because repeatability is important for debugging, the random seed can be specified with the -R flag.
Output Report∞
When the autotest script is run, it generates a JSON file which tabulates the entirety of the test results. This file can be useful for reproducing the results (particularly failures) in specific environments and machine architectures.
{
"build-info" : {},
"pass" : true,
"num_failed" : 0,
"num_checks" : 369656,
"num_warnings" : 55,
"command-line" : " ./xautotest -v -o autotest.json",
"run-mode" : "RUN_ALL",
"rseed" : 1660183820,
"stop-on-fail" : false,
"tests" : [
{"id": 0, "pass":true, "num_checks": 1, "num_passed": 1, "name":"null"},
{"id": 1, "pass":true, "num_checks": 2, "num_passed": 2, "name":"libliquid"},
{"id": 2, "pass":true, "num_checks": 4, "num_passed": 4, "name":"agc_crcf_dc_gain_control"},
{"id": 3, "pass":true, "num_checks": 3, "num_passed": 3, "name":"agc_crcf_scale"},
{"id": 4, "pass":true, "num_checks": 1, "num_passed": 1, "name":"agc_crcf_ac_gain_control"},
{"id": 5, "pass":true, "num_checks": 1, "num_passed": 1, "name":"agc_crcf_rssi_sinusoid"},
{"..."},
{"id":1091, "pass":true, "num_checks": 5, "num_passed": 5, "name":"lshift"},
{"id":1092, "pass":true, "num_checks": 5, "num_passed": 5, "name":"rshift"},
{"id":1093, "pass":true, "num_checks": 5, "num_passed": 5, "name":"lcircshift"},
{"id":1094, "pass":true, "num_checks": 5, "num_passed": 5, "name":"rcircshift"}
]
}
Autotest Examples∞
Run all autotests matching the string " firfilt ":
$ ./xautotest -s firfilt
40: firfilt_xxxf
40: firfilt_xxxf:
203 : PASS passed 8 / 8 checks (100.0%) : firfilt_rrrf_data_h4x8
204 : PASS passed 16 / 16 checks (100.0%) : firfilt_rrrf_data_h7x16
205 : PASS passed 32 / 32 checks (100.0%) : firfilt_rrrf_data_h13x32
206 : PASS passed 64 / 64 checks (100.0%) : firfilt_rrrf_data_h23x64
207 : PASS passed 16 / 16 checks (100.0%) : firfilt_crcf_data_h4x8
208 : PASS passed 32 / 32 checks (100.0%) : firfilt_crcf_data_h7x16
209 : PASS passed 64 / 64 checks (100.0%) : firfilt_crcf_data_h13x32
210 : PASS passed 128 / 128 checks (100.0%) : firfilt_crcf_data_h23x64
211 : PASS passed 16 / 16 checks (100.0%) : firfilt_cccf_data_h4x8
212 : PASS passed 32 / 32 checks (100.0%) : firfilt_cccf_data_h7x16
213 : PASS passed 64 / 64 checks (100.0%) : firfilt_cccf_data_h13x32
214 : PASS passed 128 / 128 checks (100.0%) : firfilt_cccf_data_h23x64
==================================
PASSED ALL 600 CHECKS
==================================
Run test 405 :
$ ./xautotest -t 405
405 : PASS passed 16 / 16 checks (100.0%) : demodsoft_apsk8
==================================
PASSED ALL 16 CHECKS
==================================