I wrote an article in the May 2016 issue of Visual Studio Magazine titled “Classic Stats, Or What ANOVA with R Is All About”. See https://visualstudiomagazine.com/articles/2016/05/01/anova-with-r.aspx.
In statistics because there are hundreds of common techniques (and thousands of rare techniques), one of the biggest challenges is knowing what technique to use. ANOVA, which stands for “analysis of variance” is used when you want to infer if the averages (means) of three or more groups of data are the same (or not), in situations where you only have samples of the groups.
In my article I set up a hypothetical demo problem where there are three different English classes at a large high school. Each English class uses a different textbook and a different teaching strategy. After the classes finish, if you could, you’d test all the students in the three classes, but because the hypothetical test is expensive and time consuming, you can only give the test to a few students in each class.
In the demo, the sample test scores are analyzed, giving an “F statistic” which in turn gives a “p-value” of 0.004. This is the probability that the true means of the source groups (all English students, not just the samples) are the same. Here, the p-value is small (where small means less than 0.05) so you’d conclude that the means are not all the same, or in other words, at least one of the class means is different, or in other words, maybe textbook and teaching technique have an effect.
Three important things. First, the term analysis of variance is somewhat confusing – behind the scenes, the technique uses mathematical variance to make an inference about means. Second, the results are just probabilities not certainties. Third, ANOVA makes certain math assumptions that you can’t entirely check in practice.