Reported information

Failure message, why?

When an assertion fails, a message is logged containing:

the body of the statement that failed
the name of the file and the line of the failed assertion
the name of the test case containing this assertion

The purpose of all these information is to isolate as quickly as possible the test that failed from the others. The feedback that the execution of the test case provides is an important cue, for the following reasons:

within the scheme of a continuous build/test, the logs available from the server contain this information, which points to a particular statement in the code
the cost for reproducing an error is induced by the following steps:
- identify the test module that failed in case there are many
- compile and run the test module to reproduce the error
- identify the line of the code that failed,
- fix the test directly if all the information is enough, or start a debug session

We can see from the scheme above that reproduction of an error is costly, since usually one tends to reproduce the error, which in turn induces at least the compilation of the test module. Also, a hidden cost is the lookup at the line of code that contains the failing statement, which triggers a sequence of back and forth lookup between the log on one hand and the code on the other hand.

The information extracted from the logs suggests the following fact:

	Tip
	Richness of the information contained in the logs is a key for the rapid understanding and the resolution of a failed statement

Default reporting

When an assertion fails, BOOST_TEST reports details and values on the operands of statement that lead to the failure.

Example: BOOST_TEST reporting

Code
#define BOOST_TEST_MODULE boost_test_macro3 #include <boost/test/included/unit_test.hpp> BOOST_AUTO_TEST_CASE( test_op_reportings ) { int a = 13, b = 12; BOOST_TEST(a == b); BOOST_TEST(a < b); BOOST_TEST(a - 1 < b); BOOST_TEST(b > a - 1); }

Code

#define BOOST_TEST_MODULE boost_test_macro3
#include <boost/test/included/unit_test.hpp>

BOOST_AUTO_TEST_CASE( test_op_reportings )
{
  int a = 13, b = 12;
  BOOST_TEST(a == b);
  BOOST_TEST(a < b);
  BOOST_TEST(a - 1 < b);
  BOOST_TEST(b > a - 1);
}

Output
> ./boost_test_macro3 --log_level=all Running 1 test case... Entering test module "boost_test_macro3" test.cpp(12): Entering test case "test_op_reportings" test.cpp(15): error: in "test_op_reportings": check a == b has failed [13 != 12] test.cpp(16): error: in "test_op_reportings": check a < b has failed [13 >= 12] test.cpp(17): error: in "test_op_reportings": check a - 1 < b has failed [13 - 1 >= 12] test.cpp(18): error: in "test_op_reportings": check b > a - 1 has failed [12 <= 12] test.cpp(12): Leaving test case "test_op_reportings"; testing time: 484us Leaving test module "boost_test_macro3"; testing time: 588us *** 2 failures are detected in the test module "boost_test_macro3"

Output

> ./boost_test_macro3 --log_level=all
Running 1 test case...
Entering test module "boost_test_macro3"
test.cpp(12): Entering test case "test_op_reportings"
test.cpp(15): error: in "test_op_reportings": check a == b has failed [13 != 12]
test.cpp(16): error: in "test_op_reportings": check a < b has failed [13 >= 12]
test.cpp(17): error: in "test_op_reportings": check a - 1 < b has failed [13 - 1 >= 12]
test.cpp(18): error: in "test_op_reportings": check b > a - 1 has failed [12 <= 12]
test.cpp(12): Leaving test case "test_op_reportings"; testing time: 484us
Leaving test module "boost_test_macro3"; testing time: 588us

*** 2 failures are detected in the test module "boost_test_macro3"

In the above example, the values of the operands are reported for inspection, which is more valuable as a copy of the full statement. However, we can observe that they are not treated symmetrically:

"a - 1 < b" reports "13 - 1 >= 12" failed
"b > a - 1" reports "12 <= 12" failed

More details on how the Unit Test Framework parses the statement are given in this section.

Custom messages

While perfectly exact and precise, the file name, test case name, line number of a failed statement carries an information that is partial with regards to the meaning of the failed statement. Sometimes these information are not informative enough. The BOOST_TEST macro let you override the default message by the use of a second argument, as shown on the following example.

Example: BOOST_TEST optional failure message

Code
#define BOOST_TEST_MODULE boost_test_message #include <boost/test/included/unit_test.hpp> BOOST_AUTO_TEST_CASE( test_message ) { const int a(1), b(2); BOOST_TEST(a == b, "a should be equal to b: " << a << "!=" << b); BOOST_TEST(a != 10, "value of a=" << a); }

Code

#define BOOST_TEST_MODULE boost_test_message
#include <boost/test/included/unit_test.hpp>

BOOST_AUTO_TEST_CASE( test_message )
{
  const int a(1), b(2);
  BOOST_TEST(a == b, "a should be equal to b: " << a << "!=" << b);
  BOOST_TEST(a != 10, "value of a=" << a);
}

Output
> ./boost_test_message --log_level=all Running 1 test case... Entering test module "boost_test_message" test.cpp(12): Entering test case "test_message" test.cpp(15): error: in "test_message": a should be equal to b: 1!=2 test.cpp(16): info: check 'value of a=1' has passed test.cpp(12): Leaving test case "test_message"; testing time: 219us Leaving test module "boost_test_message"; testing time: 318us *** 1 failure is detected in the test module "boost_test_message"

Output

> ./boost_test_message --log_level=all
Running 1 test case...
Entering test module "boost_test_message"
test.cpp(12): Entering test case "test_message"
test.cpp(15): error: in "test_message": a should be equal to b: 1!=2
test.cpp(16): info: check 'value of a=1' has passed
test.cpp(12): Leaving test case "test_message"; testing time: 219us
Leaving test module "boost_test_message"; testing time: 318us

*** 1 failure is detected in the test module "boost_test_message"

Boost C++ Libraries