CSE 15L, Spring 2008

Software Tools and Techniques Laboratory

The Scientific Debugging Process

We can outline the scientific debugging process in a number of steps:

0. Characterize the intended behavior of the code. Focus on what matters to a user. You don't need to explain how the program works internally; just the specific details of what it should do. Consider questions like: What input does it take? What output should it give?

For each deviation from the intended behavior:

1. Characterize a failure. A failure is a deviation from intended behavior (not a defect you notice in the code). Characterizing the failure means specifying how the behavior deviated from what was intended. Consider whether it is a compile-time, run-time, or semantic error.
2. Form a hypothesis. A hypothesis is an attempt to understand what is happening. If you've just noticed a failure, your hypothesis might speculate about conditions that lead to the failure, or might identify an apparent defect in the code that produced it. Record each hypothesis even if you discard it later; invalidating hypotheses is an important part of the process.
3. Make a prediction. While hypotheses are based on what you've already observed, predictions are what you expect to observe when you conduct an experiment. A scientific prediction is based on a hypothesis; if the prediction is false, the hypothesis is incorrect.
4. Perform an experiment. Change the conditions and compile or run the program again. You might change the conditions by giving the program different input, adding debugging output, or altering the code in some way.
5. Observe the results. What is the behavior now?
6. Reach a conclusion. Did the results match your prediction, or is your hypothesis invalidated? Do the results suggest a new hypothesis? Note: if you changed the code and the results aren't what you've predicted, you should probably undo the change!

Remember: use the scientific method to test hypotheses like the bug is fixed, this change fixed the bug, and the program behaves as intended too! These should lead to predictions, experiments, and results of their own. Don't jump to the conclusion that you've fixed the bug or that the program works without good (documented) evidence.

Note also that the process isn't usually as linear as it appears in the steps above. Each failure often leads to a sequence of hypotheses; each hypothesis often gives rise to multiple cycles of prediction, experiment, results, and conclusion. Hypotheses like the program behaves as intended don't even involve a specific failure. You may have multiple hypotheses and predictions simultaneously. This complexity is one reason you should keep detailed records as you work and try to focus on one specific problem at a time.