Productively Distracted
Posted Wednesday March 16, 2011 around 12:04 AM

Edit 2/7/2012: This is a moderately bad approach to the problem. While an interesting hack I highly discourage using this in a production environment. If you want this approach in python I recommend checking out the lettuce project for a much more sane approach.


The goal of unit testing is to isolate each part of the program and show that the individual parts are correct. A unit test provides a strict, written contract that the piece of code must satisfy. Wikipedia

For most functionality this is accomplished by writing a test that exercises your functionality covering usual usage, and maybe a few edge cases you can think of. By and large this produces code that looks something like this:

Function To Test

def is_numeric(value_in):
    try:
        float(value_in)
        return True
    except Exception:
        return False
Download

Basic Unit Test

class TestIsNumeric(unittest.TestCase):
    def test_valid_numeric_strings(self):
        self.assertTrue(is_numeric("1"))
        self.assertTrue(is_numeric("-1"))
        # etc ...

    def  test_invalid_strings(self):
        self.assertIsFalse(is_numeric("Bad String"))
        self.assertIsFalse(is_numeric("Speaks Volumes"))
        # etc ...
Download

Now there is surely nothing wrong with this code. It exercises your code and ensures the is_numeric function is working as designed. There are some issues with this approach however. First of all the same code is repeated all over the place (a clear violation of the dry principal). Second, figuring out what the actual data was that caused the test to fail requires some digging in the code after running the test suite. But even worse, is that if the line self.assertTrue(is_numeric("1")) fails, and the line self.assertTrue(is_numeric("-1")) also fails, you have lost all visibility to see that the second assert is failing. This can hide large problems when running long test suites.

Scenario Testing

To be clear, scenario testing may or may not be a real thing or even a good idea. I seem to remember seeing a python package out there that supports it, but it seemed to be a dead project. The idea behind my testing methodology is to stop writing lots of asserts hidden in tests, and instead write one well formed test that I can pass in different scenario data to via parameters.

So instead of the above tests I can write my test like this:

class TestIsNumeric(unittest.TestCase):
    __metaclass__ = ScenarioMeta

    class is_numeric_basic(ScenerioTest):
        scenarios = [
            dict(val="1", expected=True),
            dict(val="-1", expected=True),
            dict(val=unicode("123" * 9999), expected=True),
            dict(val="Bad String", expected=False),
            dict(val="Speaks Volumes", expected=False)
        ]

        def __test__(self, val, expected):
            actual = is_numeric(val)
            if expected:
                self.assertTrue(actual)
            else:
                self.assertFalse(actual)
Download

Running this from the console produces the following results:

$ python scerariotest.py -v
test_is_numeric_basic_0 (__main__.is_numeric_basic) ... ok
test_is_numeric_basic_1 (__main__.is_numeric_basic) ... ok
test_is_numeric_basic_2 (__main__.is_numeric_basic) ... ok
test_is_numeric_basic_3 (__main__.is_numeric_basic) ... ok
test_is_numeric_basic_4 (__main__.is_numeric_basic) ... ok

----------------------------------------------------------------------
Ran 5 tests in 0.001s

OK
Download

So with just a few lines of code, 5 tests were written. These are still hard coded values however. So while there is a lot less repeated code, the coverage could still be broader. Because the ScenerioTest.scenarios structure just needs to be an iterable list or callable, calculated values can be added at runtime. For example to cover unicode values we could use a list comprehension like so:

class TestIsNumeric(unittest.TestCase):
    __metaclass__ = ScenarioMeta

    class is_numeric_basic(ScenerioTest):
        scenarios = [
            dict(val="1", expected=True),
            dict(val="-1", expected=True),
            dict(val=unicode("123" * 9999), expected=True),
            dict(val="Bad String", expected=False),
            dict(val="Speaks Volumes", expected=False)
        ]
        scenarios += [(dict(val=unicode(x), expected=True),
                       "check_unicode_%s" % x) for x in range(-2, 3)]

        def __test__(self, val, expected):
            actual = is_numeric(val)
            if expected:
                self.assertTrue(actual)
            else:
                self.assertFalse(actual)
Download

The output from this would be:

$ python scerariotest.py -v
test_is_numeric_basic_0 (__main__.is_numeric_basic) ... ok
test_is_numeric_basic_1 (__main__.is_numeric_basic) ... ok
test_is_numeric_basic_2 (__main__.is_numeric_basic) ... ok
test_is_numeric_basic_3 (__main__.is_numeric_basic) ... ok
test_is_numeric_basic_4 (__main__.is_numeric_basic) ... ok
test_is_numeric_basic_check_unicode_-1 (__main__.is_numeric_basic) ... ok
test_is_numeric_basic_check_unicode_-2 (__main__.is_numeric_basic) ... ok
test_is_numeric_basic_check_unicode_0 (__main__.is_numeric_basic) ... ok
test_is_numeric_basic_check_unicode_1 (__main__.is_numeric_basic) ... ok
test_is_numeric_basic_check_unicode_2 (__main__.is_numeric_basic) ... ok

----------------------------------------------------------------------
Ran 10 tests in 0.001s

OK
Download

With just one line of code, we get 4 more tests and a lot more visibility into the test suite. The code for this isn't ready for general use yet, but I think that the idea is interesting and worth pursuing. I have posted the meta class and test code from this post as a gist on github which you can get here. We are using this for our tests at work right now. If it ends up being a good idea we will make an actual package for it.

blog comments powered by Disqus