I came across one of my statistical peeves again today, so here’s a rant: if somebody is using a survey or poll to illustrate the deep divides between groups of people, and the numbers show something like 60% of group A believe X and 60% of group B believe not-X, that does not show that groups A and B disagree. In fact, it means that almost all of group A and B agree with each other. 40% A and 40% of B believe X, and also 40% of A and 40% of B believe not-X. That’s 80% of everybody.
Let’s take the example that came up this morning: a study asked men and women if their workplace had done enough to achieve gender equality. I don’t remember the exact numbers, but it was something like: 54% of men felt that their workplace had done enough, whilst only 38% of women felt that their workplace had done enough. The person presenting those numbers did so as if it was obvious that this demonstrated a fundamental difference in how men and women viewed their workplaces—and it seemed to me that the rest of the people to whom those numbers were presented took it precisely that way. But that’s the opposite of what happened.
Let’s assume, to take our first look, that the survey had somehow normalized the size and gender make-up of workplaces, so we are mostly talking about the same thing. This is almost certainly false, but it’s the simplest, so let’s start as if it were true. And with more plausibility, let’s believe that the 38% of women who think their workplace has indeed done enough share that opinion with the 38% of men who work there. And furthermore let’s assume that the 46% of men who think their workplace has not done enough share that opinion with the 46% of women who work there. That means that 38% of men and women work at places they think have done enough—they agree. And a further 46% of men and women agree that their workplaces have not done enough—they also agree. The remainder, a mere 16%, or about one out of every six people, work at a place where the men feel that the workplace is fine, but the women feel that it is not. For five out of six of the people surveyed, the response does not differ by gender, but rather by the experience of their workplaces.
Those numbers—54% of A feel X and only 38% of not-A agree—should be obviously indicating that most women and men agree, and that it’s only the difference between those numbers that disagree.
Now, if we were to take a second look, abandoning the obviously questionable assumption that the survey had normalized the size and gender-makeup of workplaces, then things get much murkier. After all, most workplaces that I’ve been in are not close to an even breakdown. It could well be that men in male-dominated professions have very different responses than women in male-dominated professions. If, f’r’ex, some places employ five times as many women as men, and those particular places have high positive responses from their employees, then those 38% of women may be agreeing with only 7% or 8% of men about those specific workplaces. But it’s also possible that those employers are viewed as falling particularly short by their employees. Just to construct a possibility, the workplaces that are noticeably deficient actually employ 60% of all the women in the survey but only 40% of the men. If that’s the case, and if all the respondents at those places entirely agree, then the actual disagreement may be around 6% of men who work at places that they think have not done enough, while the women who work there are, unlike the men, satisfied.
Did anyone follow that?
Yeah, the second look is going to be extremely confusing, and unless you want to look at the actual results (the researchers are probably still working their way through them) you probably won’t have any way of knowing what is going on behind those big numbers. They may indicate that there is indeed a deep divide, but they may not. At any rate, without a lot more information, it isn’t going to contradict the first look, which—again, and this is my point—is showing that most people really do agree on these sorts of things.
Tolerabimus quod tolerare debemus,