Testing with saru

So everyone should be running tests on their code. We have hundreds and they're never enough. But how to run and collate results from all these tests. There are plenty of testing frameworks out there, but each one seems married to a particular language. What if parts of your code are in python, parts in C++, parts in php etc. You've been doing the right thing and using "the right tool for the job" but now you have a mish-mash of code. That was the case I was in a while ago, and I decided that I'd be better off having a testing system that could test a bunch of languages. So I wrote saru.

saru is opensource (BSD) and is the simplest testing framework I could come up with.
So how do you use it?

Heres an example test in python

#!/bin/python
# SARU : tag example
import sys
print >> sys.stderr, "Log message"
sys.exit(1)

The same thing again in C++

// SARU : tag example
#include <iostream>
int main()
{
  std::cerr << "Log message" << std::endl ;
  return 1;
}

The convention is that tests are single applications that return 1 for failure and 0 for success. To distinguish test files from other files such as mocks, fixtures or other helper code, tests are tagged with a SARU tag. Now to run these tests

saru-run-tests suite

We get the following output:

example00.py : FAILED???
==MESSAGE==
saru-run-test : execution of test failed with error code 1
==STDERR==
Log message


example01.cpp : FAILED???
==MESSAGE==
saru-run-test : execution of test failed with error code 1
==STDERR==
Log message


0 / 2

Lets change both of those files to return 0 and rerun the tests and we should get

example00.py : OK
example01.cpp : OK
2 / 2

Now this should also catch and report compilation errors in the C++. Theres a bunch of stuff not explained here that I'll detail in following posts including

1. How to make multiple tests in a single file
2. How to specify compiler options for C++
3. What would need to happen to make saru work on windows
4. How to run subsets of tests
5. How to extend saru to run other languages
6. What are these saru logs?
7. Things that still need to be done to make saru cooler