Hi everybody!
Here I talk about various tools that may be used to develop the unit tests with C++, in particular about those that we use in PDF Creator, and about the way we deploy it.
As PDF Creator evolved, it started to become more evident that there was a need for an easy-to-use testing procedure. The huge code base, lots of clients, quirks of the PDF format and of the library itself — these were the factors that turned every change in the code into a difficult task. Currently, every time we release a new version (actually, even more frequently), we pass the library through a set of visual tests. Some tests produce PDF files; their quality and correctness may be checked visually. Other tests convert EMF files, producing a visible result, too. There are also ASP and VB scripts among the tests. The set of the tests grows constantly, however, even when the library passes through all of them OK, there is no guarantee that all the functions of the library have been retained.
Because of that, while developing PDF Creator 3.9, we paid a lot of attention to the refactoring of our code. One of our refactoring goals was to improve testability of the code. After we determined the goals, we had to choose an environment for creation of the test units. Initially, it was a choice between the classic CppUnit and the Boost.Test Library.
Here is a summary of the requirements that we imposed on the testing environment:
- Writing a test should require minimal preparation.
- Test results should be clearly revealed and easily observed. Integration into Visual Studio is a plus.
- The environment must be cross platform. (In the future, we plan to make the PDF library platform independent.)
- The size of the environment must be small.
- Test units must be automatically excluded from the Release build.
I’ve had some experience with the CppUnit before, and I was not pleased. It failed to pass requirements 1, 2, and 4. It contains a lot of redundant stuff we did not actually need. The interface of CppUnit is not trivial, and we would have to add a lot of code ourselves. Negative. Especially after nUnit.
We were going to use Boost.Test, however, we discovered that we would have to include the entire Boost Library, with all its bells and whistles. I am not a big expert in Boost. If there is a way to use Boost.Test without adding the entire library to the project, please tell us how. We had to reject Boost.
Then we had to look for an alternative. I think that http://www.gamesfromwithin.com/articles/0412/000061.html is a must read. It is an excellent comparison of various test unit tools.
After reading the article, of course, we looked into CxxTest (http://cxxtest.sourceforge.net/), and it disappointed us. Documentation is huge, but not very understandable. The latest revision was outdated – from 2004! Compilation requires Perl ! Another wrong environment.
We looked into some other variants, and none of them seemed to fit. With our hope of finding anything slowly dying already, we found UnitTest++. (Applause!) You can see it here: http://unittest-cpp.sourceforge.net/. One of the UnitTest++ developers, Noel Llopis, is also the author of the excellent article referenced above. Here is his description of his own product: http://www.gamesfromwithin.com/articles/0603/000108.html
UnitTest++ fits all 5 of our requirements. It’s trivial to write a test. The environment is easy to understand and cross-platform. Whenever a test fails, it is detached into a separate project. The toolset seamlessly integrates into Visual Studio. When the tests are run, the output window displays the number of tests, elapsed time, and reviews of the failed tests. It was like a dream come true.
We started to use UnitTest++ in PDF Creator. Soon we discovered that we did not receive its excellent features “for free”. Let’s look under the hood of a typical test:
TEST(SomeTest)
{
const int expected = 123;
int res = testedFunction();
CHECK_EQUAL(res, expected);
}
During compilation, the TEST macro turns into a class named TestSomeTest, that inherits from an unknown class UnitTest::Test. The last line of the code after that looks like this:
void TestSomeTest::RunImpl(UnitTest::TestResults& testResults_) const
As you can see, the body of the test is created by RunImpl. After we looked inside the CHECK_EQUAL macro, we understood that it may render correct results only under some limited circumstances — only inside methods, defined with the Test macros. That meant that we had to forget about normal refactoring. In particular, it is impossible to separate a test method.
It is also impossible to temporarily disable a method, since it is a macro. To switch a test off, one has to comment it.
Another problem: IntelliSense doesn’t always work properly when a test method is being written. Evil consequences of the use of macros, again. It is also hard to extend and improve the functionalities of UnitTest++. Perhaps that’s the reason why it has not been updated for quite a long time – since April, 2007.
However, it is not a bad environment. Refactoring problems may be by-passed. The other problems may be put up with. In my next article, I plan to provide more details about testing with UnitTest++ and the process of hunting for memory leaks with the help of memleaks.
Vitaly Shibaev
