Data collection is a complex and critical piece of bug investigation. Data can come from many sources:

• The results of specific "experiments" such as the success or failure of a unit test.
• Observations about the state of the environment or the system made by you, your team, or by end users.
• Artifacts produced by the running system such as log files or error messages.

One common mistake when collecting data is the production of information for the sake of keeping busy or in the hope that a general fishing expedition will reveal a critical piece of information. It almost never does. Using your theory or theories (or reason for the the lack thereof as described previously) to choose specific data collections that will isolate the correct one is by far the critical skill needed for debugging. It is also the part of the scientific method that is hardest to get right, and the success of many experiments hinges on the question of whether the results ultimately support or refute the theory being tested.

Putting that problem aside for a moment, there is a lower hurdle of simply making sure that the result of any data collection at least in conception is supposed to confirm or deny a theory. This means:

1. Start by determining what pieces of information are needed, and then gearing your collections to that information specifically.
2. For each data collection, have a clear notion of possible outcomes and how each outcome maps to supporting, refuting, or is irrelevant to each theory.
3. Watch out for data collections that seem discriminating, but in fact have outcomes that support all or most of your theories. Also watch out for outcomes that would refute all of your theories, because assuming that you are a reasonably good theorizers, that outcome is probably extremely unlikely and so may not be worth doing at all.
4. If possible, talk through these outcomes and your reasoning with others on your team to help find gaps in your thinking.
5. If after this analysis you determine that a data collection is not going to provide you with good information, just don't do it. Don't convince yourself that just doing it might provide you will useful information somewhere down the line. 99% of the time you will just have extra data that serves no purpose, and may actually confuse things later on.

I'm stopping this whole "Distance Bug Investigation, Part X" thing since it makes the titles meaningless. I might even go back and change the older posts so that they have reasonable titles as well, so don't be surprised...
Tomorrow: Common Data Collection Techniques, with Caveats