Testing Basics

Motivation

• Testing software is just as important as calibrating experimental equipment, or checking the purity of reagants
  • But it's done far less often
  • Reason #1: people don't know how
  • Reason #2: low standards can perpetuate themselves until someone decides to make quality an issue
    • American cars in the 1970s
• Study after study has proved that the more time you spend testing, the less total time it takes to develop working software
  • So long as you know how to test systematically…
  • …and use the right tools to help you…
  • …and design your software to be testable
    • Just like chip designers
• Studies also show that how well a team tests is a good predictor of how good they are at meeting schedule and quality targets
  • Because discipline matters more than genius
• Comment on this slide

Terminology

• Unit testing (or component testing): testing one component in isolation
  • E.g., test that all of a class's methods do what they're supposed to
• System testing (or integration testing): testing that components work properly in combination
  • E.g., make sure that when X calls Y, it passes parameters in the right order
• These are two ends of a spectrum
  • Most classes rely on several others, and so cannot be tested in isolation
    • Often pursue a bottom-up testing strategy
    • Test the simple classes first, then the ones that use them, then the ones that use those
  • Often makes sense to test subsystems (made up of several classes), rather than the entire program
    • E.g., test the voice recognition module
  • Sometimes have to build scaffolding to support testing
    • Like the scaffolding on a building site, it's extra code that is used during the construction process, but not delivered to customers
• Regression testing: rerunning tests periodically to see whether the code has regressed
  • I.e., whether things that used to work are now broken
  • Every time a change is made to the system, the programmer making the change should rerun the regression tests before checking her changes into version control
    • Never slow other people down by breaking things that they're relying on
  • A large program that doesn't have regression tests is difficult (sometimes impossible) to maintain
    • The only way to know whether a change to one part will break something somewhere else is to run the whole program
• A fixture is a thing to be tested
  • E.g., an object, a data structure, a complete database
• A test is something you do to a fixture
  • In order for a test to be useful, you must be able to compare its actual outcome with its expected outcome
• A test result can be one of:
  • Pass: the actual outcome is what was expected
  • Fail: the actual outcome is different from what was expected
  • Error: something went wrong with the test itself (i.e., the test contains a bug)
    • Don't know anything about the system being tested
• Comment on this slide

Example: Rectangle Overlap

• Want to test a function overlap that calculates the overlap between two rectangles
  • Each rectangle is represented by an instance of the class Rect
    • Constructor takes four non-negative integer arguments (low and high XY coordinates)
    • Class has methods getLoX, getLoY, getHiX, and getHiY
    • Rectangle is guaranteed to have non-zero area
  • Function returns:
    • A new rectangle representing the overlap of its inputs if they overlap
    • None if there's no overlap
    • Note: since rectangles have to have non-zero area, touching on an edge or corner does not count as overlap
• Analysis
  • This is unit testing (one thing in isolation)
    • Assume for the moment that Rect is correct
    • I.e., that it has been tested elsewhere
  • Each fixture will be a pair of rectangles
    • The test will be to pass them to overlap, and see if the output is correct
• First version of overlap:

  •     def overlap(left, right):
            '''Calculate overlap between two rectangles.'''
            loX = max(left.getLoX(), right.getLoX())
            hiX = min(left.getHiX(), right.getHiX())
            loY = max(left.getLoY(), right.getLoY())
            hiY = min(left.getHiY(), right.getHiY())
    
            if (loX >= hiX) or (loY > hiY):
                return None
    
            return Rect(loX, loY, loX, hiY)
    

  • How many errors can you spot?
• First test passes:

  •     r1 = Rect(0, 0, 1, 1)
        r2 = Rect(2, 2, 3, 3)
        result = overlap(r1, r2)
        if  result == None:
            print 'pass'
        else:
            print 'fail'
    

    pass
    

• But not the second one:

  •     r1 = Rect(0, 0, 1, 1)
        r2 = Rect(0, 1, 3, 3)
        result = overlap(r1, r2)
        if result == None:
            print 'pass'
        else:
            print 'fail'
    

    Traceback (most recent call last):
      File "overlap.py", line 97, in ?
        result = overlap(r1, r2)
      File "overlap.py", line 64, in overlap
        return Rect(loX, loY, hiX, hiY)
      File "overlap.py", line 9, in __init__
        assert 0 <= loY < hiY, 'illegal Y coordinates'
    AssertionError: illegal Y coordinates
    

  • This is an error in the test
    • A good test will always run, no matter what state the code is in
• Revise the test and rerun:

  •     try:
            r1 = Rect(0, 0, 1, 1)
            r2 = Rect(0, 1, 3, 3)
            result = overlap(r1, r2)
            if result == None:
                print 'pass'
            else:
                print 'fail'
        except Exception, e:
            print 'error', e
    

    pass
    

  • Why is this change important?
    • Need to be able to run all tests, all the time
    • If the first test raises an exception that isn't handled, the other ninety-nine tests won't run at all
• So, what's the problem?
  • Passing loX to Rect's constructor twice, rather than loX and hiX
  • Fix and rerun tests
    • Still fails
  • Testing loY > hiY instead of loY >= hiY
  • Fix and rerun tests
    • They pass!
• We ought to test that overlap works when rectangles do overlap

  •     try:
            r1 = Rect(0, 0, 3, 3)
            r2 = Rect(1, 1, 2, 2)
            result = overlap(r1, r2)
            if result == Rect(1, 1, 2, 2):
                print 'pass'
            else:
                print 'fail'
        except Exception, e:
            print 'error', e
    

    error name 'overlap' is not defined
    

  • This is only one case out of many

    • Figure 9.1: Possible Overlap Tests

  • How many tests should we write? When should we stop?
• Comment on this slide

General Rules for Unit Tests

• Make tests independent of each other
  • No interaction between tests
    • Nothing one test does can affect any other test
    • In particular, the order in which tests are run shouldn't matter
  • If two tests depend on one another, then when the first fails, the second either:
    • Isn't run (so you have less information), or
    • Runs, but gives you the wrong answer (which is even worse than no information)
• Keep tests small
  • Each tests exactly one thing
  • When a test fails, you'll know the cause right away
• Don't hard-code absolute path names, magic numbers, etc.
  • Want to be able to move tests from machine to machine
  • Have to b able to maintain tests as code evolves
    • Test code should be at least as well written as production code
• Comment on this slide

A Simple Testing Framework

• Step back a moment and think about how we're writing our tests
  • Each has two rectangles as input, and expects either a rectangle or None
  • Each reports passes, fails, or errors
• Design principle: every time you see a pattern, try to capture it in code
  • Write the shared code once
  • Encourage people to do things the right way by making that way easiest
• Step 1: write a function that runs a test and reports its result

  • def runTest(left, right, expected):
        try:
            assert left is not None
            assert right is not None
            actual = overlap(left, right)
            if actual == expected:
                return 'pass'
            else:
                return 'fail'
        except Exception, e:
            return 'error'
    

  • Just as easy to return strings as to define constants for pass, fail, and error
  • Note: assertions on input arguments are inside the try/except
    • Passing None as the left or right rectangle is an error in the test
• Step 2: write some tests
  • Whenever possible, create a table of inputs and results
    • Easier for people to read, understand, and add to than code
    • Especially if those people aren't programmers

  • def runTest(left, right, expected):
        try:
            assert left is not None
            assert right is not None
            actual = overlap(left, right)
            if actual == expected:
                return 'pass'
            else:
                return 'fail'
        except Exception, e:
            return 'error'
    

• We can make this even more flexible by writing a function that tests an arbitrary function
  • Remember that def f(*args) creates a function that takes any number of arguments
  • If you have a tuple of values called vals, then f(*vals) matches the values with f's arguments
    • E.g., if vals is (11, 12, 13), then f(*vals) is the same as f(11, 12, 13)
    • Gives you a way to call an arbitrary function with an arbitrary set of arguments in a uniform way
  • Use this to write a function runAllTests that takes two arguments:
    • The function you actually want to test
    • A list of lists, each of which consists of zero or more arguments, and the expected result

  • def runAllTests(func, tests):
        pass = fail = error = 0
        for testCase in tests:
            try:
                args, expected = testCase[:-1], testCase[-1]
                actual = func(*args)
                if actual == expected:
                    pass += 1
                else:
                    fail += 1
            except Exception, e:
                error += 1
    
        for (label, count) in (('pass', pass), ('fail', fail), ('error', error)):
            print '%s: %4d' % (label, count)
    
    runAllTests(overlap, tests)
    

  • Advantage: reusability
    • Can use this to test any function
  • Disadvantage: obscurity
    • The extra level of abstraction makes it harder to understand what's going on
    • But you only have to understand one hard thing once, instead of many slightly less hard things over and over
• Take a look in the next lecture at a full-fledged unit testing framework
• Comment on this slide

Choosing Test Cases

• You can't test everything
  • There are roughly 10 million 7-digit phone numbers
  • A function that puts two phone numbers in order has 10¹⁴ possible inputs
  • At a million tests a second, it will take 10⁸ seconds (or roughly three years) to check them all
    • And then we move on to the second function…
  • Faster computers won't get us out of this hole
    • How many possible inputs are there to a function that sorts a list of strings?
• Human beings are creatures of habit
  • Tend to make the same kinds of errors over and over again
  • So test for those first
  • Has an interesting side effect: once you start testing for habitual errors, programmers become more conscious of them, and make them less often
• The first tests you should write:
  • Tests you expect to succeed
    • Boundary cases (e.g., sort the empty list, or a list of one value)
    • Simplest interesting case (e.g., sort a list of two values)
    • General case (e.g., sort a list of nine values)
    • If duplicate values are allowed, make sure you test with them
  • Tests you expect to fail
    • Invalid input (e.g., passed a dictionary instead of a list)
    • Remember, error handling is part of the interface too
  • Sanity tests
    • Make sure data structures remain consistent
    • If there is redundant information, check it against itself
• A catalog of errors
  • Numbers: zero, largest, smallest magnitude, most negative
  • Structures: empty, exactly one element, maximum number of elements
    • Duplicate elements (e.g., the letter "J" appears three times in a string)
    • Aliased elements (e.g., a list contains two references to another list)
    • Circular structures (e.g., a list that contains a reference to itself)
  • Searching: no match found, one match found, multiple matches found, everything matches
    • Code like x = findAll(structure)[0] is almost always wrong
    • Should also check aliased matches (same thing found multiple times)
• Comment on this slide