Chapter 6 Testing on example

We learned how to write simple tests to check if our functions are working as expected. We saw that testing helps catch mistakes early and gives us confidence that our code is correct.

In this tutorial, we’ll take testing one step further by looking into white- and black-box testing on a specific example.

6.1 Function

Let’s start with a simple function that checks whether a student passes a course.

To pass, a student needs to meet certain requirements:

  • At least 60% on the exam
  • At least 75% attendance

However, there’s one exceptional case to reward high-performing students who may have slightly lower attendance:

  • if a student scores 90% or more on the exam, they can still pass with at least 60% attendance.

Here’s how we might write a function to check for this:

def is_passing(attendance_percentage, exam_percentage):
    """
    Determines passing based on exam and attendance percentage.
    """
    result = False
    if attendance_percentage >= 75:
        if exam_percentage >= 60:
            result = True  # default case  
    elif attendance_percentage >= 60 and exam_percentage >= 90:
        result = True  # exceptional case 
    return result

6.2 White-box testing

Now that we have our function, we can start thinking about how to test it properly. We’ll begin by checking that the function runs and gives us the correct result in a simple, straightforward case. After that, we’ll gradually add more test cases as we explore different types of test coverage. Each new type will help us catch more potential issues and make our testing more thorough.

It’s possible to write a few tests and feel like everything is fine, but if we’re not careful, we might miss some important situations where our code could fail. Therefore, we need a systematic approach for defining test cases. We’ll walk through our example function and explore different types of coverage, like function coverage, statement coverage, decision coverage and condition coverage. These might sound fancy, but don’t worry–we’ll explain each one clearly, with examples, and show how they help us build stronger, more reliable programs.

Let’s see how to make our testing even better! For an effective learning experience, it’s best to try the solution on your own and do the recommended exercises.

6.2.1 Function Coverage

The first and simplest type of test coverage is function coverage. This just means: Did we run the function at all?

If we call the function at least once–no matter what the input is–we’ve achieved function coverage. It’s like checking if the lights in any room work by flipping the switch. If the light turns on or off, at least we know the wiring is connected!

For our is_passing() function, we may test a situation where the student has both enough attendance and a passing exam score. In this situation, the function should return True:

"""
>>> is_passing(75, 60) # default case
True
"""

6.2.2 Statement Coverage

Now that we’ve seen function coverage–just making sure the function runs–it’s time to go a little deeper.

Statement coverage checks whether every line of code (or statement) in the function has been executed at least once during testing. It’s like checking if the lights in all rooms work by flipping switches. It only matters that each switch was touched at least once.

To cover all the statements, we need to make sure both the “default case” and the “exceptional case” are tested. We should add an additional test case (*) to ensure that every line of code is executed at least once:

"""
>>> is_passing(75, 60) # default case
True
>>> is_passing(60, 90) # exceptional case (*)
True
"""
Look at the function and check which lines are executed by the first and which by the second test case. 
def is_passing(attendance_percentage, exam_percentage):
    result = False                                              # 1, 2
    if attendance_percentage >= 75:                             # 1, 2
        if exam_percentage >= 60:                               # 1
            result = True  # default case                       # 1
    elif attendance_percentage >= 60 and exam_percentage >= 90: #    2
        result = True  # exceptional case                       #    2
    return result                                               # 1, 2

6.2.3 Decision Coverage

Now that we’ve covered statement coverage, we’ll go even further with decision coverage.

Decision coverage looks at whether every decision point in the code (like an if statement) has been tested both ways–True and False. It’s like checking that in each room a light switch has been flipped to its both positions: ‘on’ (when the switch is flipped up) and ‘off’ (when the switch is flipped down).

In our is_passing() function, there are three key decision points:

  1. The first if statement checks if the student’s attendance is greater than or equal to 75.
  2. The inner if statement checks if the student scored at least 60% in the exam score.
  3. The elif statement checks if the student has at least 60% attendance and 90% exam score.

So far, we have not tested at all the cases when students fails. To achieve decision coverage, we need to add test cases (*) to test each conditions both ways, which means we’ve checked that the program behaves correctly in all decision situations:

"""
>>> is_passing(75, 60) # default case 
True
>>> is_passing(75, 59) # default case not met (*) 
False
>>> is_passing(60, 90) # exceptional case
True
>>> is_passing(60, 89) # exceptional case not met (*) 
False
"""

Look at the function and identify which lines are executed by each of the test cases (1–4 as above).
For lines with conditions, check how those conditions evaluate. You may use the source code of the function or its diagram.

Solution. 

def is_passing(attendance_percentage, exam_percentage):
    result = False                                              # 1,   2,   3,   4
    if attendance_percentage >= 75:                             # 1 T, 2 T, 3 F, 4 F
        if exam_percentage >= 60:                               # 1 T, 2 F
            result = True  # default case                       # 1
    elif attendance_percentage >= 60 and exam_percentage >= 90: #           3 T, 4 F
        result = True  # exceptional case                       #           3
    return result                                               # 1,   2,   3,   4

6.2.4 Condition Coverage

The last level that we will explore is condition coverage.

Condition coverage goes one step further by testing each individual condition inside a decision. For example, if we have an if statement with two conditions connected by and, we want to test each condition separately to make sure both are working as expected. It’s like checking each switch that controls the same light independently in both its ‘on’ and ‘off’ positions.

We have four conditions in our function (grouped and sorted):

  • attendance_percentage >= 75
  • attendance_percentage >= 60
  • exam_percentage >= 90
  • exam_percentage >= 60

In the decision coverage, we treated the last two together as they are in the same if statement. Now, we need to look at them separately and define two additional test cases (*):

"""
>>> is_passing(75, 60) # default case
True
>>> is_passing(75, 59) # default case not met
False
>>> is_passing(60, 90) # exceptional case 
True
>>> is_passing(60, 89) # exceptional case not met 
False
>>> is_passing(59, 90) # exceptional case not met due to attendance (*)
False
>>> is_passing(59, 89) # exceptional case not met due to attendance and exam (*)
False
"""
Look at the all four conditions and all six test case. Create a table illustrating for each test case (column) how particular conditions (rows) evalute (cell). Hint: use arguments as headers of colums.

Solution. To keep the table more compact, we simplified variable names.

Condition 75,60 75,59 60,90 60,89 59,90 59,89
attendance >= 75 True True False False False False
attendance >= 60 True True True True False False
exam >= 90 False False True False True False
exam >= 60 True False True True True True

Now, with these six test cases, we can be pretty sure that our function works in every possible scenario.

6.3 Black-box testing

So far, we’ve looked at how programs work by reading the code. But what if we want to check if a program works without seeing the code? That’s where black-box testing comes in. In this method, we test a program just by giving it input and checking the output. We don’t need to know how it works on the inside–we just need to know what it’s supposed to do.

Let’s take another look at the rules for is_passing():

  • A student needs at least 75% attendance and at least 60% on the exam to pass a course.
  • High-performing students with at least 90% exam score and at least 60% attendance also pass.

We can imagine these rules as a set of simple checks. Let’s look at how we might test them without seeing the code. To make this easier, we can draw a picture–a graph that shows which combinations of scores and attendance result in a pass or a fail. This is called a decision chart or test space diagram.

In this diagram:

  • The x-axis shows attendance (from 0% to 100%),
  • The y-axis shows exam scores (also from 0% to 100%),
  • We mark pass areas in green and fail areas in red.

By looking at this picture, we can easily see which combinations pass or fail. This helps us plan which inputs to test. Try draw it yourself.

Solution.

Now let’s focus on the edges–the places where the outcome changes from pass to fail or fail to pass. These are called boundaries, and testing them is called boundary value analysis.

Why test the boundaries? Because errors often happen right at the edge of a rule.

Our cut-off values are 60, 75, and 90. For example:

  • What happens if a student has exactly 75% attendance?
  • What if they have 74% (just one percent less)?
  • What if they score exactly 60% on the exam?
  • What if they get 59%?

By testing values just below, at, and just above the rule, we can make sure the program handles the limits correctly.

Now we want to test all the important spots–the ones right around the decision boundaries.

Think of it like checking all the corners in a room to make sure everything is safe. In our case, we’re checking all the ‘corners’ in the diagram where the decision might change. Add critical points to your plot and check your result with our solution.

Solution.

Next, we can try to reduce the number of test cases without losing important checks. Some test cases are very similar–like testing two values that are both clearly passing or both clearly failing. These cases usually don’t add much new information, so we can skip them and test these spots indirectly.

For example, take a look at the following test cases:

"""
>>> is_passing(59, 90) # A 
False
>>> is_passing(59, 89) # B is indirectly tested 
False
>>> is_passing(60, 89) # C
False
"""

In this example, we can skip test case B, because it is already covered by the other two tests. The attendance_percentage in B is the same as in A, and the exam_percentage in B is the same as in C. Since those values are already tested, running B again doesn’t give us any new information.

So, go back to your plot and choose only the most useful test cases. Then compare your selected cases with our solution to see if you’ve covered the important spots.

Solution.

After going through this process, you’ll notice that the test cases you picked are very similar to the ones you’d choose with white-box testing–even though you never looked at the code! If your test cases look a little different from ours, don’t worry. That’s totally normal. With more practice, you’ll get a better sense of which test cases matter most and how to choose them wisely.