Why use saru for testing?

I've had a few questions about why I use my own little testing framework, saru, for testing, rather than using something standard like cxxunit or whatever.

One of the key problems is testing across multiple languages. We use C++, PHP and python for various pieces of our pipeline. Saru was designed so that plugging in a new language is easy. So if someday we need to support Java then I'm not worried. Now we could use cxxunit for C++, phpunit for php, etc, but I like to have unified reporting and a bit more integration of these tests.

Another problem I have with using a pure C++ framework is sometimes I want to test the condition "Class Foo should not be default constructible." The easiest way for me to check that is to have a fragment of code that should fail to compile with given error messages.  You can't test that with a C++ testing framework. (In this particular case you may be able to do something with SFINAE style template hackery, or fork and call a compiler, but these just feel very hackish)

My third reason for using saru is that sometimes I want to do things that are not easily expressed in C++, but easily expressed in other languages. Things like "are the source files correctly formatted?", or "do all C++ exceptions have a corresponding PHP handler?", or "are there tests for each of the classes in this directory?". These are all C++ related questions, but much easier expressed in bash using awk/sed/grep or python than in pure C++.

So thats why I use saru.

What's new in saru

So to support some of our testing infrastructure I've added some features to saru. The ones of note are:

Support for skipped tests. These are tests that don't run for whatever reason. In our case it was some tests that aren't fully implemented, that we didn't want showing up as fails. Skipped tests now show up in their own count, and tests can have sub tests that are skipped.

Logging of test history into a sqlite database. Now all test results and test output is stored into a sqlite database. We use this to create pass/fail charts for the tests. This has been invaluable in tracking down regressions and intermittent failures.

Multiple Tests in a single saru file

Testing with saru is supposed to be easy to do in any language.
So the interface to support multiple tests from a single file has to be easy...

All you need do is print the right stuff to stdout and stderr and return the right value and you're done.

In the single test case all that mattered was the return value, everything else was just informational in the case of failure.

So what should the output look like to make multiple tests work... well here's a sample.

STDOUT

test_00_dummy_pass: OK
test_01_dummy_fail: FAILED
1/2

STDERR

<@saru start test_00_dummy_pass @>
Some info about the dummy_pass test
<@saru end test_00_dummy_pass @>
<@saru start test_01_dummy_fail @>
This test fails 
And so this message will appear in the test output
<@saru end test_01_dummy_fail @>

Whipping up a python script that prints these outputs and running it through

saru-run-test suite .

gives the following results

test.py::test_00_dummy_pass : OK
test.py::test_01_dummy_fail : FAILED???
==MESSAGE==

==STDERR==
This test fails 
And so this message will appear in the test output


1 / 2

Admitadly the output is not that pretty, and the mechanism is not prefectly robust.
But it has done everything I need from running multiple tests from a single file.

Of course you'd probably want to write a helper library to get that outputting correct.
And to help you use "good testing practices" like fixtures.
Some of these helper libraries already exist. The C++ one is already part of saru, as a pure header. There is a python helper library that will be added shortly, and a PHP library that is in development.

I might look at how to use the C++ library in a future post.

Testing with saru

So everyone should be running tests on their code. We have hundreds and they're never enough. But how to run and collate results from all these tests. There are plenty of testing frameworks out there, but each one seems married to a particular language. What if parts of your code are in python, parts in C++, parts in php etc. You've been doing the right thing and using "the right tool for the job" but now you have a mish-mash of code. That was the case I was in a while ago, and I decided that I'd be better off having a testing system that could test a bunch of languages. So I wrote saru.

saru is opensource (BSD) and is the simplest testing framework I could come up with.
So how do you use it?

Heres an example test in python

#!/bin/python
# SARU : tag example
import sys
print >> sys.stderr, "Log message"
sys.exit(1)

The same thing again in C++

// SARU : tag example
#include <iostream>
int main()
{
  std::cerr << "Log message" << std::endl ;
  return 1;
}

The convention is that tests are single applications that return 1 for failure and 0 for success. To distinguish test files from other files such as mocks, fixtures or other helper code, tests are tagged with a SARU tag. Now to run these tests

saru-run-tests suite

We get the following output:

example00.py : FAILED???
==MESSAGE==
saru-run-test : execution of test failed with error code 1
==STDERR==
Log message


example01.cpp : FAILED???
==MESSAGE==
saru-run-test : execution of test failed with error code 1
==STDERR==
Log message


0 / 2

Lets change both of those files to return 0 and rerun the tests and we should get

example00.py : OK
example01.cpp : OK
2 / 2

Now this should also catch and report compilation errors in the C++. Theres a bunch of stuff not explained here that I'll detail in following posts including

1. How to make multiple tests in a single file
2. How to specify compiler options for C++
3. What would need to happen to make saru work on windows
4. How to run subsets of tests
5. How to extend saru to run other languages
6. What are these saru logs?
7. Things that still need to be done to make saru cooler