Expected failures specification

While in a perfect world all test assertions should pass in order for a test module to pass, in some situations it is desirable to temporarily allow particular tests to fail. For example, where a particular feature is not implemented yet and one needs to prepare a library for the release or when particular test fails on some platforms. To avoid a nagging red box in regression tests table, you can use the expected failures feature.

This feature allows specifying an expected number of failed assertions per test unit. The value is specified during test tree construction, and can't be updated during test execution.

The feature is not intended to be used to check for expected functionality failures. To check that a particular input is causing an exception to be thrown use BOOST_CHECK_THROW family of testing tools.

The usage of this feature should be limited and employed only after careful consideration. In general you should only use this feature when it is necessary to force a test module to pass without actually fixing the problem. Obviously, an excessive usage of expected failures defeats the purpose of the unit test. In most cases it only needs be applied temporarily.

You also need to remember that the expected failure specification is per test case. This means that any failed assertion within that test case can satisfy the expected failures quota. Meaning it is possible for an unexpected failure to occur to satisfy this quota.

	Note
	If an assertion at fault is fixed and passed, while an expected failures specification still present, the test case is going to fail, since the number of failures is smaller than expected.

Usage with manually registered test cases

To set the value of expected failures for the manually registered test unit pass it as a second argument for the test_suite::add call during test unit registration.

Example 21. Expected failures specification for manually registered test case

#include <boost/test/included/unit_test.hpp>
using namespace boost::unit_test;

//____________________________________________________________________________//

void free_test_function()
{
    BOOST_CHECK( 2 == 1 );    
}

//____________________________________________________________________________//

test_suite*
init_unit_test_suite( int, char* [] ) {
    framework::master_test_suite().
        add( BOOST_TEST_CASE( &free_test_function ), 2 );

    return 0;
}

//____________________________________________________________________________//

Source code | Show output

> example --log_level=message
Running 1 test case...
test.cpp(8): error in "free_test_function": check 2 == 1 failed
Test case has less failures then expected

*** No errors detected

Usage with automatically registered test cases

To set the number of expected failures for the automatically registered test unit use the macro BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES before the test case definition.

BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES(test_case_name, number_of_expected_failures)

You can use this macro both on a file scope and inside a test suite. Moreover you can use it even if name of test units coincide in different test suites. Expected failures specification applies to the test unit belonging to the same test suite where BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES resides.

Example 22. Expected failures specification for automatically registered test case

#define BOOST_TEST_MODULE example
#include <boost/test/included/unit_test.hpp>

//____________________________________________________________________________//

BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES( my_test1, 1 )

BOOST_AUTO_TEST_CASE( my_test1 )
{
    BOOST_CHECK( 2 == 1 );
}

//____________________________________________________________________________//

BOOST_AUTO_TEST_SUITE( internal )

BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES( my_test1, 2 )

BOOST_AUTO_TEST_CASE( my_test1 )
{
    BOOST_CHECK_EQUAL( sizeof(int), sizeof(char) );
    BOOST_CHECK_EQUAL( sizeof(int*), sizeof(char) );
}

//____________________________________________________________________________//

BOOST_AUTO_TEST_SUITE_END()

Source code | Show output

> example --report_level=short
Running 2 test cases...
test.cpp(10): error in "my_test1": check 2 == 1 failed
test.cpp(21): error in "my_test1": check sizeof(int) == sizeof(char) failed [4 != 1]
test.cpp(22): error in "my_test1": check sizeof(int*) == sizeof(char) failed [4 != 1]

Test suite "example" passed with:
  3 assertions out of 3 failed
  3 failures expected
  2 test cases out of 2 passed