CSE 15L, Winter 2009

Scientific Debugging

The Scientific Debugging Process

We can outline the scientific debugging process in a number of steps:

0. Characterize the intended behavior of the code. Focus on what matters to a user. You don't need to explain how the program works internally; just the specific details of what it should do. Consider questions like: What input does it take? What output should it give?

For each deviation from the intended behavior:

1. Characterize a failure. A failure is a deviation from intended behavior (not a defect you notice in the code). Characterizing the failure means specifying how the behavior deviated from what was intended. Consider whether it is a compile-time, run-time, or semantic error.
2. Form a hypothesis. A hypothesis is an attempt to understand what is happening. If you've just noticed a failure, your hypothesis might speculate about conditions that lead to the failure, or might identify an apparent defect in the code that produced it. Record each hypothesis even if you discard it later; invalidating hypotheses is an important part of the process.
3. Make a prediction. While a hypothesis is based on what you've already observed, a prediction is what you expect to observe when you conduct an experiment. A scientific prediction is based on a hypothesis; if the prediction is false, the hypothesis is incorrect.
4. Perform an experiment. Change the conditions and compile or run the program again. You might change the conditions by giving the program different input, adding debugging output, or altering the code in some way.
5. Observe the results. What is the behavior now? Results are observations, not interpretations. "The program printed 42" is an observation. "The program worked" is an interpretation, and a vague one at that.
6. Reach a conclusion. A conclusion is an interpretation. Did the results match your prediction, or is your hypothesis invalidated? Do the results suggest a new hypothesis? Note: if you changed the code and the results aren't what you've predicted, you should probably undo the change!

Remember: use the scientific method to test hypotheses like the bug is fixed, this change fixed the bug, and the program behaves as intended too! These should lead to predictions, experiments, and results of their own. Don't jump to the conclusion that you've fixed the bug or that the program works without good (documented) evidence.

Note also the process isn't as linear as it appears in the steps above. Of course a program may have multiple failures. Each failure often leads to a sequence of hypotheses, and each hypothesis should give rise to multiple cycles of prediction, experiment, results, and conclusion. Focus on one step at a time and keep detailed records as you work, as descibed in the instructions for collaborative lab notes.