Edit 2/7/2012: This is a moderately bad approach to the problem. While an interesting hack I highly discourage using this in a production environment. If you want this approach in python I recommend checking out the lettuce project for a much more sane approach.
The goal of unit testing is to isolate each part of the program and show that the individual parts are correct. A unit test provides a strict, written contract that the piece of code must satisfy. Wikipedia
For most functionality this is accomplished by writing a test that exercises your functionality covering usual usage, and maybe a few edge cases you can think of. By and large this produces code that looks something like this:
Function To Test
def is_numeric(value_in):
try:
float(value_in)
return True
except Exception:
return False
Basic Unit Test
class TestIsNumeric(unittest.TestCase):
def test_valid_numeric_strings(self):
self.assertTrue(is_numeric("1"))
self.assertTrue(is_numeric("-1"))
# etc ...
def test_invalid_strings(self):
self.assertIsFalse(is_numeric("Bad String"))
self.assertIsFalse(is_numeric("Speaks Volumes"))
# etc ...
Now there is surely nothing wrong with this code. It exercises your code and ensures the is_numeric function is working as designed. There
are some issues with this approach however. First of all the same code is repeated all over the place (a clear violation of the
dry principal). Second, figuring out what the actual data was that caused the
test to fail requires some digging in the code after running the test suite. But even worse, is that if the line
self.assertTrue(is_numeric("1")) fails, and the line self.assertTrue(is_numeric("-1")) also fails, you have lost all visibility
to see that the second assert is failing. This can hide large problems when running long test suites.
Scenario Testing
To be clear, scenario testing may or may not be a real thing or even a good idea. I seem to remember seeing a python package out there that supports it, but it seemed to be a dead project. The idea behind my testing methodology is to stop writing lots of asserts hidden in tests, and instead write one well formed test that I can pass in different scenario data to via parameters.
So instead of the above tests I can write my test like this:
class TestIsNumeric(unittest.TestCase):
__metaclass__ = ScenarioMeta
class is_numeric_basic(ScenerioTest):
scenarios = [
dict(val="1", expected=True),
dict(val="-1", expected=True),
dict(val=unicode("123" * 9999), expected=True),
dict(val="Bad String", expected=False),
dict(val="Speaks Volumes", expected=False)
]
def __test__(self, val, expected):
actual = is_numeric(val)
if expected:
self.assertTrue(actual)
else:
self.assertFalse(actual)
Running this from the console produces the following results:
$ python scerariotest.py -v
test_is_numeric_basic_0 (__main__.is_numeric_basic) ... ok
test_is_numeric_basic_1 (__main__.is_numeric_basic) ... ok
test_is_numeric_basic_2 (__main__.is_numeric_basic) ... ok
test_is_numeric_basic_3 (__main__.is_numeric_basic) ... ok
test_is_numeric_basic_4 (__main__.is_numeric_basic) ... ok
----------------------------------------------------------------------
Ran 5 tests in 0.001s
OK
So with just a few lines of code, 5 tests were written. These are still hard coded values however. So while there is a lot less repeated
code, the coverage could still be broader. Because the ScenerioTest.scenarios structure just needs to be an iterable list or callable, calculated values can
be added at runtime. For example to cover unicode values we could use a list comprehension like so:
class TestIsNumeric(unittest.TestCase):
__metaclass__ = ScenarioMeta
class is_numeric_basic(ScenerioTest):
scenarios = [
dict(val="1", expected=True),
dict(val="-1", expected=True),
dict(val=unicode("123" * 9999), expected=True),
dict(val="Bad String", expected=False),
dict(val="Speaks Volumes", expected=False)
]
scenarios += [(dict(val=unicode(x), expected=True),
"check_unicode_%s" % x) for x in range(-2, 3)]
def __test__(self, val, expected):
actual = is_numeric(val)
if expected:
self.assertTrue(actual)
else:
self.assertFalse(actual)
The output from this would be:
$ python scerariotest.py -v
test_is_numeric_basic_0 (__main__.is_numeric_basic) ... ok
test_is_numeric_basic_1 (__main__.is_numeric_basic) ... ok
test_is_numeric_basic_2 (__main__.is_numeric_basic) ... ok
test_is_numeric_basic_3 (__main__.is_numeric_basic) ... ok
test_is_numeric_basic_4 (__main__.is_numeric_basic) ... ok
test_is_numeric_basic_check_unicode_-1 (__main__.is_numeric_basic) ... ok
test_is_numeric_basic_check_unicode_-2 (__main__.is_numeric_basic) ... ok
test_is_numeric_basic_check_unicode_0 (__main__.is_numeric_basic) ... ok
test_is_numeric_basic_check_unicode_1 (__main__.is_numeric_basic) ... ok
test_is_numeric_basic_check_unicode_2 (__main__.is_numeric_basic) ... ok
----------------------------------------------------------------------
Ran 10 tests in 0.001s
OK
With just one line of code, we get 4 more tests and a lot more visibility into the test suite. The code for this isn't ready for general use yet, but I think that the idea is interesting and worth pursuing. I have posted the meta class and test code from this post as a gist on github which you can get here. We are using this for our tests at work right now. If it ends up being a good idea we will make an actual package for it.