Do equations show that large scale conspiracies will quickly reveal themselves?

I’m currently reading Ben Goldacre’s I think you’ll find it’s a bit more complicated than that, and had just read the piece about research showing that we tend to overlook holes in evidence if the evidence supports our preconceptions, when a link to an article claiming “Large scale conspiracies would quickly reveal themselves, equations show” appeared in my Facebook feed. “Physicists decided to test whether some science-related conspiracies alleged to exist were in fact tenable. The answer – they’d all have given themselves away in less than four years”, the article claimed. “Well, that supports my preconceptions”, I thought. “I wonder whether there are any holes in the evidence”.

Well, there are certainly some holes in the Science Daily reporting of the work – “physicists” turned out to be only one physicist, Dr. David Grimes, he turned out to have considered more than just science-related conspiracies (although he does state a particular interest science-related ones), and he theorized three closely related models, not just one as reported by Science Daily. But the original paper is freely available online, and is easily readable for those of us not daunted by a few statistical distributions. And it’s in a peer-reviewed publication, too, which is encouraging (though not as carefully proof-read as it might have been – I was amused by the suggestion of “a small devious cohort of rouge scientists”).

Dr. Grimes suggests that the chance of conspiracies being exposed by whistle blowers, as a function of time, is likely to follow one of three distributions, the choice of which depends on factors such as whether new people are recruited to the conspiracy and whether existing members are “rapidly removed due to internal friction or otherwise” (the plot of quite a few murder mysteries).

He then examined three conspiracies that have been exposed, used those to calibrate his model, then used the calibrated model to estimate how long it would take for various conspiracies to be exposed – that the moon-landings were faked, climate-change is a hoax, vaccination is dangerous and that a cure for cancer is being suppressed by vested interests.

I can see a few problems with this that could possibly be fixed, but one problem that I think holes the work beneath the waterline and sinks it completely.

Let’s look at some of the relatively fixable issues first.

1. Dr Grimes has calibrated his models, but he hasn’t verified them. The claim that the exposure of conspiracies will follow these models is purely theoretical and unconfirmed, and because of (3) below is likely to be over-simplistic.
2. Three sample points is nowhere near enough to calibrate three models, even if they are closely related, particularly as it’s not clear why those particular cases were chosen.
3. The models take no account of social differences in conspiracies, such as how cohesive the group of (supposed) conspirators are, how severe the consequences would be for a whistle-blower, and so on. Credit to Dr. Grimes – he does acknowledge this, noting that the model ” does not consider the dynamics, motivations and interactions of individual agents”. I would add “or groups of agents”, to cover the case where disparate communities – media, police and politicians, for example, or Templars, Rosicrucians and Freemasons – are presumed to have colluded in the conspiracy. There are good statistical techniques for dealing with these sorts of biases, which I expect Dr. Grimes knows as “a physicist and cancer researcher at Oxford University”, but if he needs any help I’m sure his fellow Guardian journalist Dr Ben Goldacre could give him some pointers.
4. The work seems to overestimate the effect of exposure. Dr. Grimes recognizes that “[t]he grim reality is that there appears to be a cohort so ideologically invested in a belief that for whom no reasoning will shift, their convictions impervious to the intrusions of reality” but actually a plausible response of the conspiracy theorists is that the conspiracies have been exposed (after all, how else would the conspiracy theorists know about them?) but the whistle-blowers have been discredited or ignored. And that response can’t be dismissed out of hand: in the case of the Hillsborough Disaster the conspiracy to cover up culpability for the deaths of 96 people and injuries to a further 766 persisted from 1989 to 2012 despite the refusal of many people with first-hand knowledge of what happened (the Liverpool fans present) to participate in the conspiracy from the outset. Zero-day whistle-blowers, if you like.

The fatal flaw, though, seems to be that the work assumes conspiracies that have been exposed are representative of all conspiracies. Again, to Dr. Grimes credit, he acknowledges this as an issue, but claims that because of pessimistic estimates elsewhere the resulting bias is not significant (unfortunately, without some way to estimate how likely exposure of conspiracies actually is there’s no way to tell whether the pessimism of Dr. Grimes’ estimates swamp the selection bias or the selection bias swamps the pessimism of the estimates) and because “even relatively small conspiracies (such as Watergate, for example) have historically been rapidly exposed” (but how can he know how many relatively small conspiracies have not been uncovered?).

But “how likely exposure of conspiracies actually is” is what we’re trying to find out. If we use it as an input we just get our assumption fed back to us as a result – Dr. Grimes’ argument is circular. If it is the case that most conspiracies are not exposed, the sampling bias could be severe, and a more accurate calibration of his model would give far lower likelihoods of conspiracies being exposed (or make such calibration impossible, because how could you get useful data on conspiracies that have not been exposed?) In short, if you assume that conspiracies are very likely to be exposed, Dr. Grimes’ methodology will tell you that conspiracies are very likely to be exposed, because the pessimism will swamp the sampling bias. If you assume that conspiracies are very unlikely to be exposed, Dr. Grimes’ methodology will tell you they are very unlikely to be exposed (or at very least, will not tell you you’re wrong), because the sampling bias will swamp the pessimism. The methodology simply feeds your preconceptions back to you as results, and so tells us nothing.

Well, not nothing: it tells us what Dr Grimes’ preconceptions are. They’re the same as mine. Which at least gives me a little bit of echo-chamber gratification, but isn’t much practical use.