Planned vs. unplanned contrasts

A concise and clear explanation from Wendell Kimper:

Let’s say there are 4 pairwise comparisons we could possibly do on a given data set.  In Scenario A, we’ve dutifully thought through our theoretical implications, and we decide to do two comparisons, because those are the ones that are relevant to our theoretical model.  We still need to correct for the fact that we did two of them, but we don’t need to correct for the fact that there are 4 that we could have done.

In Scenario B, we didn’t plan any pairwise comparisons.  Maybe we’re not irresponsible researchers, but our data surprised us in some way that merits looking at things we didn’t expect to have to look at.  We look at our data, and see that there seem to be pretty big differences, numerically, for two of the pairwise comparisons, so we go ahead and do those tests.  We still need to correct for all 4 of the possible contrasts, even if we only did two.

In Scenario A, there’s no element of chance in the decision about which comparisons to make, but there is in Scenario B.  The differences we saw in the data in Scenario B could have been there by chance, and a different sample would have led us to a different set of pairwise comparisons.  So we have to correct for all the comparisons we could have possibly done.

Leave a Reply

Your email address will not be published. Required fields are marked *