Effect of wording on outcome of referendum

Question

In this famous episode of "Yes, Prime Minister", Sir Humphrey explains how it is possible to get two opposite results to the same opinion poll, by changing the wording and the framing. I would like to know to what extent this phoenomenon exists in referendums? My guess is that in a referendum it would be quite minor, since voters would spend more time in deciding what to vote, and not be influenced by framing effects.

I understand that it is hard to get data on this, as most referendums are done once, using the same framing, to all voters (there are no "A/B tests"). But maybe someone has found a methodology to test this question empirically: what is the potential effect of wording on the outcomes of a referendum?

Answer 1

The majority opinion among political scientists appear to be that the wording does not matter that much (framing is a different matter). At least not in a typical Western context, where most referendums are posed as yes/no questions and where referendums are held at most once every few years and preceeded by lengthy public debate.

Rocher and Lecours tackles this question - "Does the wording of a referendum question matter?" - in the book The Routledge Handbook to Referendums and Direct Democracy. They conclude:

The case studies discussed above suggest that paying attention to the clarity of a referendum question might be of limited benefit for at least three reasons. The first is that the nature of the referendum question can be a reflection of complex, sometimes long-standing, constitutional preferences, which may not be simplified easily. [...] The second reason is that the meaning of the question is provided, to a very large extent, by the political debate that surrounds the path towards the referendum campaign, as well as the campaign itself. [...] The third reason why attention paid to the clarity of a referendum question might have limited benefits is that even when the question is very short and apparently straightforward, debate about the meaning of its key terms, such as independence, cannot (and probably should not) be avoided. [...]

In the end, the voters will decide based on their understanding of the political issue on which they must take a stand. The wording of the question turns out to be relatively secondary.

Political science professor Matt Qvortrup echoes the sentiment in a BBC interview about Brexit:

He says the question is not vitally important because people will know before they enter the polling place what their views are on Europe.

He says: "The overall conclusion one can draw if one looks around the world is that question itself extremely rarely has an impact on the outcome of the referendum."

One of the more confusing referendum questions in recent years came from the 2015 Greek bailout referendum:

Should the plan of agreement, which was submitted by the European Commission, the European Central Bank and the International Monetary Fund in the Eurogroup of 25.06.2015 and is comprised of two parts that constitute their unified proposal be accepted?

The first document is entitled “Reforms For The Completion Of The Current Program And Beyond” and the second “Preliminary Debt Sustainability Analysis.”

But it appears that the Greeks understood the question well-enough and, as far as I can tell, the result has not been challenged on grounds of the question being confusing. The referendum question for the 1998 Good Friday agreement was quite confusing too - "Do you support the Agreement reached at the multi-party talks on Northern Ireland and set out in Command Paper 3883?" - but in the end, it didn't matter much.

So while you can find hundreds examples of politicians concerning themselves about exact wordings on referendums, their worries are probably unfounded.

Polls, on the other hand, are well-known to be susceptible to wording choices. And perhaps you can make the argument that wording matters (much) more on multiple choice or ranking questions.