...one of the most highly
regarded and expertly designed C++ library projects in the
world.
— Herb Sutter and Andrei
Alexandrescu, C++
Coding Standards
The Unit Test Framework offers a number of ways to run only a subset of all test cases registered in the test tree.
Each test unit (either test case or test suite) has an associated default run status. It can assume one of the three values:
Initially, the master test suite has default run status set to true. All other test units have default run status set to inherit. This implies that, unless any additional configuration is applied, all tests are designated to be run.
You can set a different default run status in any test unit by using decorators: disabled
, enabled
and enable_if
. The default run status
is set once, upon testing program initialization, and cannot be changed.
The disabled tests are not executed by default, but are still present in
the test tree, and are listed along with other tests when you use command-line
argument list_content
.
Code |
---|
#define BOOST_TEST_MODULE decorator_20 #include <boost/test/included/unit_test.hpp> namespace utf = boost::unit_test; const bool io_implemented = true; const bool db_implemented = false; BOOST_AUTO_TEST_SUITE(suite1, * utf::disabled()) BOOST_AUTO_TEST_CASE(test_1) { BOOST_TEST(1 != 1); } BOOST_AUTO_TEST_CASE(test_2, * utf::enabled()) { BOOST_TEST(2 != 2); } BOOST_AUTO_TEST_CASE(test_io, * utf::enable_if<io_implemented>()) { BOOST_TEST(3 != 3); } BOOST_AUTO_TEST_CASE(test_db, * utf::enable_if<db_implemented>()) { BOOST_TEST(4 != 4); } BOOST_AUTO_TEST_SUITE_END() |
Output |
---|
> decorator_20 Running 2 test cases... test.cpp(18): error: in "suite1/test_2": check 2 != 2 has failed [2 == 2] test.cpp(24): error: in "suite1/test_io": check 3 != 3 has failed [3 == 3] *** 2 failures are detected in the test module "decorator_20" > decorator_20 --list_content suite1* test_1 test_2* test_io* test_db |
Additionally, it is possible to statically associate a test unit with a condition. This associated condition is evaluated immediately before executing the test unit. If the condition is met, the test unit is executed. Otherwise, the test unit is skipped. It is possible to add two dependencies:
depends_on
.
precondition
.
Static configuration of the test-case filtering is used by default, unless
command-line filtering is applied. With command-line argument run_test
it is possible to alter
the static pre-set in a number of ways:
Term 'absolute' in this context means that the default run status of the
test units, which has been statically set up, is completely ignored and the
tests to be run are specified manually from scratch. First, in order to learn
what test units are registered in the test tree the user needs to use command-line
argument list_content
. Next, in order to
specify a set of test cases, the user needs to use command-line argument
run_test
with absolute value:
> test_program --run_test
=<test_set>
The format of <test_set>
value can assume a number of forms. Given
the following program:
#defineBOOST_TEST_MODULE
example #include <boost/test/included/unit_test.hpp> using boost::unit_test::label
;BOOST_AUTO_TEST_CASE
(test_1, *label("L1")) {} BOOST_AUTO_TEST_CASE(test_2, *label("L1")) {}BOOST_AUTO_TEST_SUITE
(suite_1) BOOST_AUTO_TEST_SUITE(suite_1) BOOST_AUTO_TEST_CASE(test_1) {} BOOST_AUTO_TEST_CASE(test_2) {} BOOST_AUTO_TEST_SUITE_END() BOOST_AUTO_TEST_SUITE(suite_2) BOOST_AUTO_TEST_CASE(test_1, *label("L2")) {} BOOST_AUTO_TEST_CASE(test_2, *label("L2")) {} BOOST_AUTO_TEST_SUITE_END() BOOST_AUTO_TEST_CASE(test_1, *label("L1")) {} BOOST_AUTO_TEST_CASE(test_2) {} BOOST_AUTO_TEST_CASE(test_2A) {} BOOST_AUTO_TEST_SUITE_END()
The following table illustrates how different values of <test_set>
control which test cases ware run.
Description |
Parameter value |
Test cases run |
---|---|---|
Run single top-level test case by name |
|
test_1 |
Run single nested test case by name |
|
suite_1/suite_1/test_1 |
Run single test suite by name |
|
suite_1/suite_2/test_1 suite_1/suite_2/test_2 |
Run multiple test units that are siblings of the same test suite |
|
suite_1/suite_2/test_1 suite_1/suite_2/test_2 suite_1/test_1 |
Run multiple test units that are not necessarily siblings |
|
suite_1/test_1 test_1 |
Run all tests matching to a given label |
|
test_1 test_2 suite_1/test_1 |
Run every test case in the test tree |
|
test_1 test_2 suite_1/suite_1/test_1 suite_1/suite_1/test_2 suite_1/suite_2/test_1 suite_1/suite_2/test_2 suite_1/test_1 suite_1/test_2 suite_1/test_2A |
Run every test unit in a given suite with a given prefix |
|
suite_1/test_1 suite_1/test_2 suite_1/test_2A |
Run every test unit in a given suite with a given suffix |
|
suite_1/suite_1/test_1 suite_1/suite_1/test_2 suite_1/test_1 |
Run every test unit in a given suite with a given infix |
|
suite_1/suite_2/test_1 suite_1/suite_2/test_2 suite_1/test_2 suite_1/test_2A |
Run test(s) with given name in any N-level suite |
|
suite_1/suite_1/test_2 suite_1/suite_2/test_2 |
For the syntax productions describing the structure of <test_set>
value see here.
While using manual absolute test case specification ignores the default run
status, it does not ignore the dynamic test dependencies. If test unit B
depends on test unit A
and test B
is specified to
be run by run_test
, A
is also run, even if it is not specified, and its failure may cause the execution
of B
to be skipped. Similarly,
the failed check of the precondition
may cause the test
selected test to be skipped.
Code |
---|
#define BOOST_TEST_MODULE decorator_21 #include <boost/test/included/unit_test.hpp> namespace utf = boost::unit_test; namespace tt = boost::test_tools; tt::assertion_result fail(utf::test_unit_id) { tt::assertion_result ans(false); ans.message() << "precondition failed"; return ans; } BOOST_AUTO_TEST_CASE(test_1) { BOOST_TEST(false); } BOOST_AUTO_TEST_CASE(test_2, * utf::depends_on("test_1")) { BOOST_TEST(true); } BOOST_AUTO_TEST_CASE(test_3, * utf::precondition(fail)) { BOOST_TEST(true); } |
Output |
---|
> decorator_21 --log_level=test_suite --run_test=test_2,test_3 Including test case test_1 as a dependency of test case test_2 Running 3 test cases... Entering test module "decorator_21" test.cpp(14): Entering test case "test_1" test.cpp(16): error: in "test_1": check false has failed test.cpp(14): Leaving test case "test_1"; testing time: 3ms test.cpp(26): Test case "test_3" is skipped because precondition failed test.cpp(20): Test case "test_2" is skipped because dependency test case "test_1" has failed Leaving test module "decorator_21"; testing time: 17ms *** 1 failure is detected in the test module "decorator_21" |
Term 'relative' in this context means that the configuration is based on
either the default run status of the test units or by the command-line override
specified by the absolute specification; and atop of
this, we additionally either enable some disabled test units or disable some
enabled tests units. The relative specification is controlled by command-line
argument run_test
, with the value using similar
syntax as in the absolute specification, but preceded with either character
'!'
for disabling enabled test
units or with character '+'
for
enabling the disabled test units. This can be summarized with the following
table:
command |
specification type |
semantics |
---|---|---|
> test_program --
|
disabler |
Enabled test units that match |
> test_program --
|
enabler |
Disabled test units that match |
The enabler specification is used to enable a set of test units which are initially disabled.
Code |
---|
#define BOOST_TEST_MODULE decorator_22 #include <boost/test/included/unit_test.hpp> namespace utf = boost::unit_test; BOOST_AUTO_TEST_CASE(test_1) { BOOST_TEST(true); } BOOST_AUTO_TEST_CASE(test_net, * utf::disabled() * utf::description("requires network")) { BOOST_TEST(true); } |
Output |
---|
> decorator_22 --list_content test_1* test_net : requires network > decorator_22 --log_level=test_suite Running 1 test case... Entering test module "decorator_22" test.cpp(6): Entering test case "test_1" test.cpp(6): Leaving test case "test_1" Leaving test module "decorator_22"; testing time: 5ms *** No errors detected > decorator_22 --log_level=test_suite --run_test=+test_net Running 2 test cases... Entering test module "decorator_22" test.cpp(6): Entering test case "test_1" test.cpp(6): Leaving test case "test_1"; testing time: 1ms test.cpp(13): Entering test case "test_net" test.cpp(13): Leaving test case "test_net"; testing time: 1ms Leaving test module "decorator_22"; testing time: 16ms *** No errors detected |
Conversely, the disabler specification is used to disable a set of test units which are initially enabled.
Code |
---|
#define BOOST_TEST_MODULE decorator_23 #include <boost/test/included/unit_test.hpp> namespace utf = boost::unit_test; BOOST_AUTO_TEST_CASE(test_1) { BOOST_TEST(true); } BOOST_AUTO_TEST_CASE(test_net, * utf::description("requires network")) { BOOST_TEST(true); } |
Output |
---|
> decorator_23 --list_content test_1* test_net*: requires network > decorator_23 --log_level=test_suite Running 2 test cases... Entering test module "decorator_23" test.cpp(6): Entering test case "test_1" test.cpp(6): Leaving test case "test_1" test.cpp(12): Entering test case "test_net" test.cpp(12): Leaving test case "test_net" Leaving test module "decorator_23"; testing time: 14ms *** No errors detected > decorator_23 --log_level=test_suite --run_test=!test_net Running 1 test case... Entering test module "decorator_23" test.cpp(6): Entering test case "test_1" test.cpp(6): Leaving test case "test_1" Leaving test module "decorator_23"; testing time: 5ms |
If there are both an enabler and disabler on one command line that specify the same test, the test becomes disabled. I.e., the disabler takes the precedence over the enabler.
Note | |
---|---|
While enabler additionally enables the upstream dependencies (introduced
with decorator |