This was neither up-to-date nor a poll. And not a single participant voiced their support for Hard Brexit

You really cannot blame the feeding frenzy of Breitbart, The Sun, Express, Mail, Guido Fawkes, Leave.EU and many others for their excitement over this. But it’s almost all wrong.

Not 20,000 people. Not one participant actually said they support Hard Brexit and the data is not polling data. Nor even current. How the hell could something that was meant to be, and is, an innovative academic study go so badly wrong?

The story is as follows. An academic research group trying to unpick attitudes to Brexit using novel methods shared an exclusive snapshot of its work with Buzzfeed before peer review. The data is about presenting people with randomised combinations of Brexit options – a group of eight options as a bundle versus another group of eight options as a bundle – to see what individual elements are deal-breakers.

Is it immigration, European Court of Justice, EU laws, the divorce bill, trade tariffs, payments for access, border security, etc? And how much of each is acceptable?

‘The headline conclusions are therefore that the public is surprisingly willing to compromise on some aspects of the negotiations’ is the summary the authors themselves published in an extensive LSE blog. Apart from immigration controls and ECJ authority for Leavers and continued rights of EU citizens for Remainers, there’s a lot of opinion flexibility. But this interesting and quality blog was published two days after it had given its exclusive to Buzzfeed.

The Buzzfeed article was sensationalist and led with unqualified conclusions about overall preference for Hard Brexit that were not even mentioned in the authors’ clarifying blog.

The lead conclusions touted by Buzzfeed and widely circulated thereafter were that a huge authoritative, scientific new piece of research on 20,000 people shows nearly 70% of the public, including a majority of Remainers, would prefer no deal to Soft Brexit. In a chart within the piece, the option of ‘all must leave’ for EU citizens scores 29% for Remainers and 42% for Leavers.

The Independent then followed up with a piece saying that 29% of Remainers are now quite happy to round up and deport all three million non-Brit EU citizens. The Sun were gleeful and Jacob Rees-Mogg provided a quote to them pouring scorn on the disconnect of the ‘metropolitan elite’. Guido Fawkes say ‘Remoaners have lost their battle to derail Brexit as the public overwhelmingly backs a ‘hard’ exit from the EU’. On BBC’s Sunday Breakfast News, I am informed, these findings were presented as strong evidence that most people in the UK now support a Hard Brexit.

Only that all this was neither up-to-date nor a poll. And not a single participant voiced their support for Hard Brexit. Herein lies the dangers of mixing slow-burn academic research in with the regular fast turnover of public polling.

The first thing that you should know is that the data was collected on April 26-27. This is not a new poll in August, but a photograph in time from one week after Theresa May called the snap election. In that era, May was riding high in the polls, Corbyn was on a minus 42 favourability rating, a landslide for May’s Hard Brexit mandate was widely expected and the words ‘strong and stable’ ruled the airwaves. No-one in England had heard of the DUP. May herself thought, at that time, that the country was uniting behind Brexit and it would drive her to triumph. Her theory was put to the test six weeks later.

Also the initial traction for Buzzfeed’s exclusive was carried by the report of it being a study of 20,000 people. This was not true. It was 19,758 data-points taken from 3,293 participants. Although Buzzfeed corrected this it was too late by then. Many reports still carry the 20,000 figure.

Does this invalidate the research itself? Of course not. But it does help explain, in combination with the tags of LSE, Oxford and new data, why this went viral and was taken as the absolute authoritative word on the current British opinion.

Do 29% of Remainers really want to deport all EU citizens? No, not in real life. There is a huge fault of data presentation here, as well as an issue of methodology. I am a scientist and I found the Buzzfeed charts confusing – until I got to see the methodology which the authors themselves published two days later.

Then it became clear that the numbers, from 0% to 100%, were akin to the attractiveness of the option when it appears in a bundle of options. So, 0% means it is a perfect deal-breaker in any situation and 100% is the reverse. Therefore 50% is a perfect ‘meh’.

So this number cannot be interpreted as a percentage of people supporting the option. As several commentators have pointed out since – if the scale were changed to 0 in the middle, with negative scores for undesirable items and positive scores for desirable items, like a politician’s ‘favourability’ rating, then it all becomes clear.

Are 70% really for a Hard Brexit? The only way you can accurately assess whether people support Soft Brexit, Hard Brexit, no Brexit, or Remain – is to ask them. Therefore the ‘70% back Hard Brexit’ conclusion is very wrong in my view – and it is interesting to note that the extensive LSE Blog piece on the study avoids all mention of the Soft versus Hard versus no deal comparison.

The basic methodology is a useful and very interesting one. However, the whole debacle shows exactly why you should not publish conclusions in the press before checking those conclusions are agreed by your scientific peers. For a start – reviewers would likely have caught the 0%-100% scale problem and asked for a presentation around zero to avoid ambiguity. As to the conclusions about individual items, it is interesting to see the relative public acceptability and this method is good for deciphering that, I believe. But the options must be realistic. Paying £1bn to access science and structural funds is unrealistic. Deporting all non-Brit EU citizens is wildly unrealistic and also inappropriate. An interesting lesson learned is the relative apathy towards many items. Of particular worrying note is that the most popular item among Leave voters with regard to the Irish border was ‘full passport control, full customs control’ – arguably not a politically savvy solution.

The conclusions on Hard Brexit, Soft Brexit or no deal are a different matter. Any argument that these bundles, as presented, map perfectly onto public choice on Hard, Soft or no-deal Brexit is inherently false. Again, I am relieved to see the authors completely dropped this part from their LSE blog summary.

What to do now the horse has bolted? The story that 70% are now for Hard Brexit is out. It’s chasing down rabbit holes into pubs and family conversations where technical corrections cannot easily reach. It has erroneously shifted understandings and will damage debate for months. The tale should serve as a warning to researchers and responsible journalists alike. In a fast-paced world where academics are encouraged to make impact and journalists gain more by being fast than being right, we can all get tripped up by the post-truth world. Ironically, the Buzzfeed author responsible for the piece is the author of a new book just out entitled: ‘Post-Truth: How Bullshit Conquered The World.’

Dr Mike Galsworthy (@MikeGalsworthy) is co-founder of Scientists for EU (@Scientists4EU)