Intuitive Statistics for CHI Practitioners: Developing Understanding and Avoiding Bloopers

Keywords

INTENDED AUDIENCE

THE TUTORIAL

The instructors help participants develop intuitions by beginning with simple probability (using concrete examples such as drawing marbles from urns and tossing coins), moving on to what it means to take measurements, and discussing what it takes to convince others that your results mean what you say they mean. From this basis, the tutorial proceeds slowly to statistical tests, always keeping the simple probability and measurement examples and the notion of "convincing evidence" in sight. In so doing, participants will develop intuitive notions of: probability, measures of central tendency and deviation, probability distributions, statistical power, how to design empirical studies, and how to recognize faulty designs and analyses. The instructors will also describe typical statistical "bloopers," using examples culled from the popular press and the CHI literature as well as made-up examples. Tutorial participants will be given an opportunity to take and summarize measurements, design hypothetical experiments plan the corresponding data analyses, and, if time permits, evaluate experimental analyses reported by others.

The tutorial will cover the following topics:

• Why conduct experiments? Why use statistics?
• Simple examples of using statistics.
• Simple probability, with concrete examples.
• Summary statistics.
• Simple statistical tests.
• Common errors in using statistics.

After completing this tutorial, participants will:

• Know why statistics are used, when to use them, and when not to.
• Have an intuitive grasp of statistical concepts, e.g.: probability, central tendency and deviation, probability distributions, hypothesis testing, statistical power.
• Understand at a common-sense level how to design simple but convincing experiments.
• Understand a small set of simple statistical tests that will be useful to them as CHI practitioners.
• Be able to evaluate the methods used by other practitioners in published reports; in particular, be able to recognize common errors in applying statistics.
• Be a more critical reader of supposedly "scientific" results reported in the popular press.

PROBLEMS OF TRADITIONAL STATISTICS PEDAGOGY

Some statistics courses use a highly mathematical approach, deriving -- or asking students to derive -- the formulae that underlie probability distributions or that define decision statistics. We call this the "derivational" approach. The derivational approach is supposed to build deep understanding of statistical analysis and experimental design. A typical homework assignment in a class that used this approach would not be to perform a statistical analysis on some data, but rather to prove, say, that analysis of variance is a special case of linear regression. While dragging students through lengthy and abstract derivations may work for students who plan to become statisticians or mathematicians, it doesn't work for those who just want to know how to use statistics in their job-domain. In such courses, the majority of students spend much of the time with their eyes glazed over. Since much of what is presented passes over their head, they fail to develop the deep understanding that the derivational approach is supposed to produce, and what little they do learn does not prepare them for designing real-world experiments and performing statistical analyses on the results.

The other common approach is commonly referred to as the "cookbook" approach. Instructors who use the cookbook approach skip the derivations and cut to the chase, i.e., the formulae needed to compute various decision statistics. Formulae are given in forms that expedite computation and are accompanied by examples that show how the formulae are used. This approach is obviously intended for practitioners, not statisticians. Deep understanding is definitely not the goal of the cookbook approach. Indeed, the basic assumption is that one need not understand statistics in order to use them. While it is true that one need not understand statistics as thoroughly as people who develop new statistical analyses do, some understanding is required to know what is appropriate for the problem at hand, how to interpret results correctly, and how to avoid violating assumptions underlying the formulae. These days, the cookbook approach is often accompanied by instruction in the use of the instructor's favorite statistical analysis software, so students can even avoid learning the formula for a statictical test, the conditions and assumptions underlying its correct use, etc. All they "need" learn is how to operate the software. Students often emerge from such courses believing that any set of numbers entered into the software will yield a meaningful result. But, of course, garbage in, garbage out. So whereas students who learned statistics by the derivational approach tend to avoid statistics for the rest of their lives, students who learned it by the cookbook approach produce a lot of garbage.

In addition to having their own specific problems, the derivational and cookbook methods of teaching statistics share the problem of overemphasizing complex, high-power analyses that are rarely needed by CHI practitioners. Statistical software exacerbates this by fostering analytic overkill: applying a statistical cannon to a gnat-sized problem. This may be appropriate for the domains of psychology and communications, where the problems encountered often require complex designs and analyses, and the effects sought are often subtle. However, CHI practitioners -- those of us who conduct usability tests of products or prototypes -- rarely, if ever, need complex analyses like multiple regression, analysis of co-variance, factor analysis, auto-correlation, or time-series analysis. Even three-way analysis of variance and non-linear regression are borderline. Furthermore, we usually don't care about small effects: if effects and differences aren't strong enough to practically jump out of the data, they are unimportant. Therefore, we can stick to analysis that detect only strong effects, which are, by and large, simpler. The bottom line is that for most of what we do, we can get by with a small repertoire of simple, easy-to-understand tests, so we don't need much of what is taught in statistics classes.

INSTRUCTOR BACKGROUNDS

Robin Jeffries has degrees in mathematics, computer science, psychology and Russian. She has studied and taught statistics both as a statistician and a psychologist. She did her graduate work in psychology at the University of Colorado and Post-doctoral work both at Colorado and at Carnegie-Mellon University. She was a researcher in HCI at Hewlett-Packard Laboratories for 11 years, and is currently User Interface Architect for the Developer Products Business Unit of Sun Microsystems. She has published in a variety of areas, including comparisons of user interface evaluation methods and cognitive studies of computer programers. She has taught statistics to students from grade schoolers to college graduates, including co-teaching an earlier version of this tutorial at HP Labs.