C++ Unit Tests

Hi everybody!

Here I talk about various tools that may be used to develop the unit tests with C++, in particular about those that we use in PDF Creator, and about the way we deploy it.

As PDF Creator evolved, it started to become more evident that there was a need for an easy-to-use testing procedure. The huge code base, lots of clients, quirks of the PDF format and of the library itself — these were the factors that turned every change in the code into a difficult task. Currently, every time we release a new version (actually, even more frequently), we pass the library through a set of visual tests. Some tests produce PDF files; their quality and correctness may be checked visually. Other tests convert EMF files, producing a visible result, too. There are also ASP and VB scripts among the tests. The set of the tests grows constantly, however, even when the library passes through all of them OK, there is no guarantee that all the functions of the library have been retained.

Because of that, while developing PDF Creator 3.9, we paid a lot of attention to the refactoring of our code. One of our refactoring goals was to improve testability of the code. After we determined the goals, we had to choose an environment for creation of the test units. Initially, it was a choice between the classic CppUnit and the Boost.Test Library.

Here is a summary of the requirements that we imposed on the testing environment:

  1. Writing a test should require minimal preparation.
  2. Test results should be clearly revealed and easily observed. Integration into Visual Studio is a plus.
  3. The environment must be cross platform. (In the future, we plan to make the PDF library platform independent.)
  4. The size of the environment must be small.
  5. Test units must be automatically excluded from the Release build.

I’ve had some experience with the CppUnit before, and I was not pleased. It failed to pass requirements 1, 2, and 4. It contains a lot of redundant stuff we did not actually need. The interface of CppUnit is not trivial, and we would have to add a lot of code ourselves. Negative. Especially after nUnit.

We were going to use Boost.Test, however, we discovered that we would have to include the entire Boost Library, with all its bells and whistles. I am not a big expert in Boost. If there is a way to use Boost.Test without adding the entire library to the project, please tell us how. We had to reject Boost.

Then we had to look for an alternative. I think that http://www.gamesfromwithin.com/articles/0412/000061.html is a must read. It is an excellent comparison of various test unit tools.

After reading the article, of course, we looked into CxxTest (http://cxxtest.sourceforge.net/), and it disappointed us. Documentation is huge, but not very understandable. The latest revision was outdated – from 2004! Compilation requires Perl ! Another wrong environment.

We looked into some other variants, and none of them seemed to fit. With our hope of finding anything slowly dying already, we found UnitTest++. (Applause!) You can see it here: http://unittest-cpp.sourceforge.net/. One of the UnitTest++ developers, Noel Llopis, is also the author of the excellent article referenced above. Here is his description of his own product: http://www.gamesfromwithin.com/articles/0603/000108.html

UnitTest++ fits all 5 of our requirements. It’s trivial to write a test. The environment is easy to understand and cross-platform. Whenever a test fails, it is detached into a separate project. The toolset seamlessly integrates into Visual Studio. When the tests are run, the output window displays the number of tests, elapsed time, and reviews of the failed tests. It was like a dream come true.

We started to use UnitTest++ in PDF Creator. Soon we discovered that we did not receive its excellent features “for free”. Let’s look under the hood of a typical test:

TEST(SomeTest)
{
const int expected = 123;
int res = testedFunction();
CHECK_EQUAL(res, expected);
}

During compilation, the TEST macro turns into a class named TestSomeTest, that inherits from an unknown class UnitTest::Test. The last line of the code after that looks like this:

void TestSomeTest::RunImpl(UnitTest::TestResults& testResults_) const

As you can see, the body of the test is created by RunImpl. After we looked inside the CHECK_EQUAL macro, we understood that it may render correct results only under some limited circumstances — only inside methods, defined with the Test macros. That meant that we had to forget about normal refactoring. In particular, it is impossible to separate a test method.

It is also impossible to temporarily disable a method, since it is a macro. To switch a test off, one has to comment it.

Another problem: IntelliSense doesn’t always work properly when a test method is being written. Evil consequences of the use of macros, again. It is also hard to extend and improve the functionalities of UnitTest++. Perhaps that’s the reason why it has not been updated for quite a long time – since April, 2007.

However, it is not a bad environment. Refactoring problems may be by-passed. The other problems may be put up with. In my next article, I plan to provide more details about testing with UnitTest++ and the process of hunting for memory leaks with the help of memleaks.

Vitaly Shibaev

10 comments

Skip to comment form

    • Dietrich on February 20, 2008 at 7:40 am

    Vitaly, that’s an interesting article. I would like to know more about Refactoring, I am a novice in this area. I use Extract method and Move method, Extract Class, Decompose Conditional, and Rename Method. Which methods do you use for Refactoring? How long do you work using Refactoring?
    Dietrich

  1. Hi Dietrich!

    First you need always remember main goal of refactoring – it is code clarity. Look at your code and ask yourself – how I can make it more clear? Imagine resulted code and think over process to achieve that result, divide it on elementary steps, such as Extract method, Rename method etc. Each step must not change program behavior, so unit tests is very useful. With experience you will make it automatically, without remembering names of Fowler’s refactorings.
    Undoubtedly the most frequently used refactoring is Extract method (with good method name).

    I advise you necessarily read these books (if didn’t read as yet) – “Refactoring” (M. Fowler) and “Code Complete” (S. McConnell).
    Concerning me – I permanently refactor my code since reading Fowler’s book, about 1.5 years.

    Best regards,
    Vitaliy Shibaev!

    • Dietrich on April 22, 2008 at 7:00 am

    Of course, I’ve read these things. Code Complete is really great. It is always at my fingertips. I see you have been dealing with refactoring for a long time. It seems you are a guru in the issue J.

    I’m a Java developer. After reading of Fowler I use JUnit Tests in my projects. I think that a self tested code is important for refactoring. So your notes are interesting for C++ developers.

    Vitaly, how much time (percentagewise) do you deal with refactoring in development process? Do you think that refactoring is creative work or it’s rather a routine for developers?

  2. Hi Dietrich!

    We aren’t divide refactoring and code writing – always try to write good new code. Because our library contains some bad code from old-old version sometimes we only refactor code all day long.
    Refactoring (in aggregate with architecture projecting) is creative work undoubtedly. See how the code is becoming better – it’s delicious 🙂
    Of course sometimes needs some routine actions, it’s normal.

    Best regards,
    Vitaliy Shibaev!

    • Bertrand Dechoux on April 25, 2008 at 6:41 am

    Hi Vitaliy!

    I found your blog entry while searching what would be the best option for unit testing in C++. I already knew about cppunit and Boost test but I was quite lost in the ‘C++ Unit Testing Framework Jungle’. The links you provided seems a good starting point. Thanks.

    I thought I would drop a comment because of this sentence “One of our refactoring goals was to improve testability of the code”. I understand your point but I am used to read it the other way around. In “test driven development”, the first step is testing and testing is a way to improve design. It seems like your team is not using this method of designing software but you might want to know more about it.

    regards

    Bertrand Dechoux

  3. Hi Bertrand!

    Thanks for your letter!
    Of course, we know TDD principles. But there was such trouble – the project existed before us and it was not contain unit tests. Some tests for PDF creation and conversion existed but it was not unit tests. Code was not very good and unit tests writing “as is” wasn’t possible for the most part. When we reorganize code unit testing became possible. Now we are writing tests before coding in the main (depends on mood and tasks 🙂 )

    Best regards,
    Vitaliy Shibaev!

  4. I’m curious, what tools do you use for your refactoring work. I use Visual Assist X but am always on the lookout for more and better tools.

    John

  5. Hi John!

    We use Visual Assist X too. It has several wonderful features such as method search(Alt+M), “Find reference”.
    But for refactorings automation it’s not very comfortable, I use only “Rename” and “Extract method” mostly.

    Best regards,
    Vitaliy Shibaev!

  6. Hi,

    I came by this post my accident. If you still interested, I can help you with setting up Boost.Test, if you clarify what is the problem specifically.

    Regards,

    Gennadiy

  7. Hi Gennadiy,

    Just one question – is there a way to link my project only with Boost.Test (without the other parts of Boost)?

    Best regards,
    Vitaliy Shibaev

Comments have been disabled.