Lessons from a Statistical Analysis Gone Wrong, Part 2

Last time, I told you how I had double-checked the analysis in a post that involved running the Johnson transformation on a set of data before doing normal capability analysis on it. A reader asked why the transformation didn't work on the data when you applied it outside of the capability analysis. 

I hadn't tried transforming the data that way, but if the transformation worked when performed as part of the capability analysis, it should work when applied outside of that analysis, too.

But the reader was correct. The transformation failed when applied by itself. 

When I'd performed the capability analysis with the Johnson transformation option selected, the analysis seemed fine to me. It had been a while since I'd done a capability analysis, but the graph looked okay.  

Then I remembered one of my first Minitab instructors, who told us "Always look at the session window."  So I did.  And there it was: 

Yes, the process capability analysis had been performed...on data that hadn't been transformed. I missed it. And it wasn't until a reader tried running the analysis a different way that my oversight was revealed. 

While Minitab does warn you that the transformation failed, you need to check the session window to see it. I've used Minitab and other statistical software packages for some time now, and I know that it's important to look at all of the output.

In this case, I only looked at the graph. Graphs tell you a lot, but you shouldn't rely on graphs alone. I knew this, and I usually do check Minitab's session window...but in this case, I didn't. 

While I should have checked the session window, there's another reason I missed the fact that the transformation hadn't occurred: when it comes to capability analysis, I was out of practice. 

Like most people who use Minitab, I have a wide range of responsibilities. Some involve statistics and data analysis, and many do not. I do some types of analysis far more frequently than others. Capability is one that I hadn't performed in a while. 

Given the time that passed since the last time I did a capability analysis with transformed data, I should have been more thorough in reviewing the output, shown here: 

My mistake seems obvious now: this graph contains a huge warning that the transformation failed. However, the warning lies not in what you see above, but instead in what this graph does not show.

For comparison, here's a capability report that involves a successful transformation: 

Yeah...when you see the transformation equation in the subhead of this graph, not to mention the words "After Transformation" in the data table, their absence in the earlier graph is very conspicuous.  

Thus, I missed my second opportunity to realize that the transformation had failed. Unfortunately, that meant that the analysis of the Triple Crown data wasn't valid. I felt like a fool for missing something that seems so obvious in hindsight.

You can bet that I'll remember to check the session window more vigilantly, and that I'll be quite a bit more cautious when performing analyses that I haven't done in a while.  

In fact, after realizing my mistake, I tried doing this analysis using the capability tools in the Assistant, which duly notified me that the analysis was suspect. Would that I had thought to use the Assistant, at least to double-check my results, in the first place!

I removed the post about American Pharoah from the blog. Then I wrote to the person who had caught my error, and expressed my gratitude—and chagrin—that he had noticed it.  

But it turned out I had even more lessons to learn from this failed analysis.  

Photo by Alex E. Proimos, used under Creative Commons 2.0.