Those Opinion Polls: A Reality Check

Posted byiainlawson27Posted inUncategorizedTags:

This is a guest post from Stephen Duncan. I don’t usually publish polls discussion on this site as James Kelly of Scot goes Pop usually has this covered but Stephen has put a lot of work into this and is clearly less excited than most pro Indy Commentators. Here he explains why.

Those Opinion Polls: A Reality Check

People seem to be working themselves into a lather of excitement over the recent spate of opinion polls on Scottish Independence, if the comments by regular columnists in, and myriad readers on, various articles in The National are anything to go by (see, for example, Scottish independence support at 56%, Ipsos Mori poll finds | The National). Whilst there have now been 4 consecutive polls showing a lead for YES, and 3 of them since the UK Supreme Court ruling on 23rd November that the Scottish Parliament did not have the right to legislate for an Independence referendum, it is wise to bear in mind that these results form a relatively small proportion of a much longer time-series recorded since the Independence referendum of 18th September 2014.

Polling Analysis

Statistics and numbers can be turn off for many but they are used to justify arguments and defend positions so it’s important to have as a full an understanding as possible of results published so that ‘the narrative’ and claims being made can be properly scrutinised and assessed. So in the interests of perspective, context and, indeed, accuracy I have looked at this in some detail and in different ways. Here’s what I found:

Batch analysis of survey results registered post 2014 using a simple average across all pollsters:

 # Surveys
Vintage36912
Earliest50.08%50.71%50.13%49.52%
Most Recent54.00%52.35%51.05%50.46%

This approach combines tranches of survey results from different pollsters and calculates the arithmetic average. The drawback to this is that averaging across polling firms can mask CHANGE in support picked up by each organisation due to inherent differences in methodologies employed by the different pollsters. On the other hand the statistical biases in the LEVEL of support recorded may be balanced out in the averaging process as long as no single firm dominates the series of surveys.

Conclusion:

There is no discernible change in rounded terms over the longer 12 month comparison where the YES/NO split is roughly 50:50. However, an increase in YES support is depicted in cohorts comprising fewer poll results with the earliest to most recent 3 survey comparison depicting an increase from 50% to 54%. This is clearly as a consequence of the most recently published surveys which been favourable to YES. This is also shown when a comparison is made between the 12, 9, 6 and 3 most recent polls with YES increasing from 50% to 54%, the most recent 3 surveys accounting for half of that rise.

Batch analysis of survey results registered post 2014 using a weighted by sample volume average across all pollsters:

 # Surveys
Vintage36912
Earliest50.14%50.75%50.16%49.54%
Most Recent54.02%52.37%51.17%50.70%

This differs from the first calculation in that all percentages are weighted by the (different) sample sizes involved in the various surveys and is technically a more accurate measure. However, the results are pretty much identical due to sample sizes being fairly standard – it’s usually just over 1000, yielding the normal +/- 3% margin error – which has the effect of rendering the weighting by volume effect fairly neutral.

Conclusion:

Similar to the unweighted assessment with a small and fairly insubstantial +1% uplift over the earliest to most recent comparisons containing 12 polls and a +1% rising to +4% increase for the cohorts comprising smaller number of poll results.

[Note: In all calculations, and as standard, I have excluded those that Don’t Know, Won’t Say and Won’t Vote).

One of the reasons why it is problematic to infer at this stage a ‘step increase’ in YES support based on these most recent polls is that two of the three recent surveys were conducted by polling firms with limited experience in conducting polls on support for Scottish Independence. Both Redfield & Wilton Strategies and Find Out Now who published results after the UK Supreme Court ruling recording YES at 52% and 54%, respectively, have only ever published one other poll on the subject. This means we have very limited data points for these firms with which to establish a benchmark and from which to discern any trend.

However, this is not the case for Ipsos who conducted the recent poll showing YES on 56% – and have carried out 17 surveys since the referendum – nor for other major and experienced polling firms Panelbase, Survation and YouGov who have published 50, 45 and 39 polls, respectively. These are now considered.

Looking at First 12 surveys post referendum versus most recent 12 and computing a weighted by sample volume average for each of the 4 firms that have polled across the period since September 2014:

PollsterSurveysYES%
IpsosFirst 1251.69%
IpsosLast 1252.64%
PanelbaseFirst 1247.70%
PanelbaseLast 1249.18%
SurvationFirst 1248.53%
SurvationLast 1250.37%
YouGovFirst 1248.37%
YouGovLast 1248.31%

Analysing by polling organisation has the advantage of comparing ‘apples with apples’ although differences  in methodologies can yield different levels of support from firm to firm in the results. However, it is probably the most accurate way, in my opinion, to gauge change in support across the time period.

Conclusion: 

Three out of the four polling firms show some limited increase which, taken in the round, could point to an uptick of around +1%. (Taking an arithmetic average of these is slightly distorting but given that sample sizes are similar in the polls this should only be in the neighbourhood of percentage decimal points).

The problem here is that neither Panelbase, Survation nor YouGov have posted any poll results since the UK Supreme Court pronouncement so no inference can be drawn as to what effect the ruling may have had on support for Independence.

Assessing YES support over the long term by each of the 4 firms that have polled across the period:

  Opinion Poll
Pollster# SurveysDateYES%DateYES%
Ipsos1724-30 Aug 201554.92%28 Nov-5 Dec 202252.00%
Panelbase5030 Oct-5 Nov 201450.60%7-10 Oct 202248.48%
Survation456-13 Nov 201447.52%24-28 Mar 202247.78%
YouGov3927-30 Oct 201452.13%30 Sep-4 Oct 202247.18%

On this basis three of the pollsters depict a reduction whilst one indicates that support has flat-lined. The drawback to this method is that the earliest result is based on a single poll while the latter is an average across the number of surveys conducted in the series per polling firm. (For example, Ipsos recorded 54.92% in its first post-referendum survey but the weighted by sample size average across 17 polls since 18th September 2014 is 52.00%). However, this does serve to highlight the danger of using a single snapshot as being the true measure of YES support. 

There have, of course, been significant fluctuations at different points across the period and these, together with the underlying long term level of YES are best illustrated graphically:

Conclusion

This points to a reduction in support from the immediate post-referendum boost. It is noted, however, that an uptick in YES can be detected from Ipsos, Survation and YouGov prior to the UKSC ruling perhaps indicative of the cost of living crises positively impacting YES sentiment.

[Note: In all calculations, and as standard, I have excluded those that Don’t Know, Won’t Say and Won’t Vote).

Further Observations

So, however you look at it, the picture over the longer term is a fairly static one, albeit YES has increased versus the baseline referendum actual result (44.70%). But the increase post-referendum happened immediately after the 18th September 2014, down probably to EVEL being announced the next day and ‘buyers regret’ on behalf of some NO voters.

So for The National, its columnists and commenters, the SNP and pro-SNP bloggers to refer variously to ‘surging’, ‘soaring’, ‘shooting’ and ‘sky-rocketing’ support for Independence is somewhat hyperbolic. And perhaps a bit desperate.

The reason why it’s desperate is that there is a continuing vacuum of leadership in the SNP on the constitutional question facing Scotland.

In addition we have been here before. There have been upward spikes measured at different points in the past, as the time-series for each pollster shows.

Some have said that the UK Supreme Court ruling is a ‘game changer’ and that historical polls are, therefore, not relevant. Of course historical polls do become less relevant over time and as circumstances change. However, you cannot describe something as a game changer on the basis of a 3 data points. You need many more to confirm a step change in the level and trend of support. Remember, even the run of 20 consecutive opinion polls in 2020-21 came to an end with support evaporating as the pandemic eased.

At this stage, therefore, we should be cautious especially when it is recalled that we were told by prominent political figures and The National’s authors that the Brexit vote, then the perceived better handling of the pandemic in Scotland and finally the Brexit implementation were all watersheds and that tidal wave in favour of YES would follow.

That may have happened if the momentum so generated by events had been captured by the SNP and used to drive support with a suitably aggressive and nuanced campaign. It wasn’t.

It remains to be seen if the leadership will ‘catch the wind’ this time or whether any momentum – generated by cost of living crises given added impetus by the UK Supreme Court ruling – will be lost due to prevarication, timidity and unpreparedness on behalf of the movement’s political leadership.

What we do know is that opinion polls are passing.

And Independence is permanent.

Sources

Wikipedia: https://en.wikipedia.org/wiki/Opinion_polling_on_Scottish_independence

Ipsos UK: https://www.ipsos.com/en-uk/news-and-events/news

Panelbase: https://drg.global/our-work/political-polls/

Survation: https://www.survation.com/

YouGov: https://yougov.co.uk/topics/overview

28 thoughts on “Those Opinion Polls: A Reality Check”

  1. I’m wary of the polls. My gut instinct tells me, he who pays the piper calls the tune…. all be it, given the nature of the intstument it can only be nuanced.

    The last time the polls did a big leap in favour of YES, was at the tail end of 2019… when pressure was mounting on Nikla to actually do something, an SNP conference was due in November, the last conference had resulted in the mini coup, and there was a slim hope of a VONC (among us mere mortals, who don’t read the instruction manual, as to how these things work). And Britain, really needed to keep everything ticking along until the end of January 2020 when Brexit would finally be enacted.

    The polls increased, which got Nikla over her hump, and the rallying cry for the SometimeNeverPeople went up, ‘LOOK AT THE POLLS, JUST LOOK AT THEM’.

    I said then, and I’ll say it now, at that time there had been no ‘national’ events that would account for that rise in support, and that it would get as high as 59%, but never hit the golden watermark of 60%, which the SNP leaders have deamed the starting line for campaigning.

    So now, Nikla appears to have some party problems, there’s been the appearance of a Kershuffle down in Westminster, that on scraping the wet paint looks very like, meet the new boss, same as the old.

    And hey, we get this…. just in time for a new leadership election don’t you think? A perfect way to kick the can down the road, and get everyone drawn into the party political soap opera.

    One thing that has changed…. it’s bloody cold, my water pipe has frozen because I’ve had the heater off… and combi boiler is now not working even if I wanted it to. And I’m not alone.

    Was in a well off home yesterday, and they (pensioners) were complaining about the cost of heating, had it turned down and, even more earth shattering, they had a Believe in Scotland wee book (normally they would have put it in the bin, as they are solid No Voters).

    In an Independent Scotland we can turn the Heating Back on. Let’s do it soon folks, no more fannying about.

    Also, some movement from the solid SNPers, they’ve been standing up to John Swinney at Constituency meetings. They want action, and they want it now.

    Liked by 7 people

    1. It’s time I agree for a changing of the guard, Sturgeon is spent 8 years of nothing and 6 mandates wasted. We need someone who can unite the movement instead of destroying it. A change is coming you can smell it in the air. People are sick of Westminster and Sturgeons inaction . We cannot afford to leave her in place she is a liability and a self serving lame duck.

      Liked by 4 people

  3. I’m sorry but I put no faith in polling companies, such as YouGov founded by Nadhim Zahawi, or IPSOS, and let’s not forget the UK government going to court to hide the result of indy polls conducted by them.

    When the time comes to exit this prison of a union, I trust Scots will make the right decision, Scots voted yes to leave the union in 2014, and they will do so again.

    https://www.dailyrecord.co.uk/news/politics/independence-referendum-figures-revealed-majority-5408163

    https://12ft.io/proxy?q=https%3A%2F%2Fwww.thenational.scot%2Fnews%2F23160897.ipsos-denies-secret-independence-polling-eddi-reader-tweet%2F

    Liked by 4 people

    2. I think the British government going to court has little to do with YouGov or any of the rest of the polling companies. The latter have a reputation to protect … I stand to be corrected though if you have evidence to back your claim.

      Like

  4. To be honest I’m less than excited about the polls, regardless of the analysis above, because how on God’s green Earth is the Yes vote so low given the dire state of the UK? What would the yes vote be if we had a competent independence minded government making the case?

    Liked by 6 people

    1. I guess the opinion polls must have had some impact on your thinking otherwise you wouldn’t be asking why “the Yes vote so low”.

      I do agree, though, that if the SNP/Scottish Government were focused as a priority on Independence support would surely be higher, and probably much higher.

      Liked by 2 people

      1. Well no Stephen they don’t influence MY thinking such. I supported indy LONG before it was anything other than a pipedream. I view opinion polls as indicators of current views and potential trends that must be caveated by all the usual – sampling size, statistical noise, ordering of questions, temporary fluctuations, randomness of sample (in statistical terms how well does the sample represent the underlying population or is it skewed by sample selection and/or methodology). Good ones can tell you something but bad ones… Of course nothing beats sampling the whole population!

        But yeah why are the polls so low 😦

        Liked by 1 person

      2. I agree with you that YES support is low irrespective of recent upticks.

        The polls are low because a) I do not discount them as irrelevant and b) I have an expectation as to where they should be in the circumstances. Without either of these opinions I couldn’t form that assessment.

        As to why YES’ support is little above its immediate post referendum level: my view is lack of leadership and strategy, a failure to score even one of the many open goals in the last 8 years and Independence simply not being an urgent priority for the SNP/Scottish Government.

        The signs and noises coming from the SNP since 23rd November do not auger well either when you consider a) the FM’s nebulous talk of a ‘special conference’ at some stage in ‘the new year’, b) the confused responses from her lieutenants regarding which election will be used as a ‘de facto referendum’ and whether its seats, votes or seats+votes, SNP only or pro-Independence parties and c) indications that it will only be vote for a mandate to beg for another S30 British sanctioned referendum in any event.

        Liked by 3 people

  6. I think all pro Independence folk know that the problem is the cabal at the top of the SNP and until we get rid of them we are shafted

    Liked by 4 people

  8. Anyone who has ever cold-canvassed and enquired as to an individual’s voting intentions will know that, for whatever reason, people often tell fibs. All you can rely on is their level of enthusiasm and to an extent, their body language. Even then, you can never predict whether or not they’ll turn-out to vote.

    Liked by 2 people

  9. Since this is Footy World Cup time, and watching defensive tactics boringly played out, perhaps explains the tactics of the SNP back three of Robertson at right back, Swinney at left back, and Sturgeon at centre half – aimlessly passing the ball to each other across the field – to play the tied game out. And so on to the rematch in a few years time.

    Liked by 3 people

    1. There is clearly evidence of a ‘cost of living-UK Supreme Court ruling’ effect already.

      To my mind the real question is whether the SNP/Scottish Government will act promptly and decisively to harness this or whether they will allow any momentum to peter out (yet again) through lack of action.

      Liked by 1 person

  11. Analysis of demographic trends and the central importance of British or Scottish identity from the post 2014 indyref survey indicates the broad composition of the vote for/against independence, and helps explain why ‘Yes’ remains relatively static despite Brexit and so many Westminster disasters:

    ‘Yes’ 1.6 million votes
    – comprising mainly Scots speakers who hold primarily to a Scottish identity (there were approx 1.6m Scots speakers according to 2011 census)

    ‘No’ 2.0 million votes
    Made up of 2 main groups:
    – people with British/English and other non-Scottish national identities, mostly Anglophone, approx 1 million incl extraction (this is also the most rapidly growing segment of Scotland’s population since abt 2000)
    – cultural/colonial assimilated Scots, heavily oriented toward middle class and Anglophone as the most pampered part of the colonised population who due to cultural assimilation hold to what they believe to be a ‘superior’ British AND hence ‘subordinated’ Scottish identity/values (i.e. the ‘dual’ or ‘false’ persona and ‘cultural illusion’, or ‘colonial mindset’), approx 1 million

    We might remember that peoples in self-determination conflict subjected to colonial/cultural assimilation processes tend to be linguistically divided and the Scots are nae different. Hence an independence movement always depends on the solidarity of the oppressed ethnic group, in this case subordinated and supposedly inferior ‘invalid’ Scots speakers. This also explains why independence “is a fight for a national culture” and involves “self-recovery” where one of the first things a liberated people do is grasp their “rusted tongue” as the oppressors language is found to “burn their lips”.

    https://yoursforscotlandcom.wordpress.com/2021/07/04/paper-two-the-determinants-of-independence/

    Liked by 4 people

  12. Thank you.

    The article confirms my own suspicions. I suspect the the marginal changes since 2014 are down to a variety of causes but none of them significant or even noticeable within the margin of error in polling data.

    Outwith polling data I apply simple observation. We have no active Campaign. We have no unity across the Independence movement. We have no plan setting out the roadmap. I could go on, but the simple evidence of inactivity surrounds us, so why pretend any step change has occurred?

    The simple truth is that between 2012 and 2014 the change was palatable. Neighbours, Pub talk, YES Stalls, Marches etc etc. It was a daily part of our lives. The legal issue did not exist. The Referendum was accepted by all. For 15 hours on polling day Scotland was free unless we voted NO. ( I use WE in the franchise context).

    In Sport, Business, War or Politics you need a goal, target or objective to focus drive. We had a fixed date in 2014. It provided the Countdown and driver that motivated people.

    Today we witness a “leader” tilting at windmills and being snubbed. She rebounds with “a Conference next year”, but only for the SNP. She has the National run a laughable story about a Bill submission at Westminster to change the Scotland Act.

    I don’t need polling data to tell me that we are treading water.

    A simple quote on why we are failing –
    “The problem is politics is made a sport, almost as much a sport as football. When it comes to politics, adults and politicians do more finger-pointing and play more games than children ever do. Too often are we rooting for the pride of a team rather than the good of the nation.”

    The YES movement has gone. Sturgeon puts HER team ahead of the Nation….it is that simple. It is a problem that has beset our Nation for Centuries….”I lead the biggest Clan, so I should lead the Army”

    The YES movement put the Nation first. We worked together on one objective….Independence.
    Sturgeon puts the SNP first. SHE will decide. Why are we all excluded from HER plan?

    In a recent interview after the SC ruling she said “ I do not speak for other Parties, I am the leader of the SNP”. That is the problem not the solution.

    If you are SSP, ALBA, ISP, Indy Labour voter etc you do not like being TOLD what the plan is (not that we have ever had a plan since 2014).

    Now the final part….what if the polling was sitting at 70/75% what would happen then? Westminster can and will still say NO. What is the Plan at that point? Will it be a Blackford style speech, “Scotland won’t accept this”

    The military have a saying… “Amateurs talk Strategy, Professionals talk Logistics”
    Great Chess players are several moves ahead of their opponent.
    Sturgeon is a gifted showman who reacts to events with utter surprise and another vague promise.

    Liked by 3 people

    1. Well spot on Clootie. I particularly concur with this:

      “Outwith polling data I apply simple observation. We have no active Campaign. We have no unity across the Independence movement. We have no plan setting out the roadmap. I could go on, but the simple evidence of inactivity surrounds us, so why pretend any step change has occurred?”

      And that is the whole point of the article … little has changed over 8 years when you look at it in the round. There have been moments of opportunity at various points along that time-line, none of which have been capitalised on. The momentum has always disipated because the leadership did either nothing or completely the wrong thing (like trying to get England to change its mind about Brexit)

      The pretence has allowed the blindly loyal to believe their wishful-thinking while providing cover, for the leadership to carry on as before by their kicking of the ball not so much into the long grass but deep into the Amazonian rain forest.

      Liked by 1 person

  13. Polls are mere snapshots of opinion. That opinion is media influenced. Given Scotland’s flaccid media it is a miracle independence support is where it is.
    Distractions are the usual ploy. Think UK economy, global economy, energy, that war in east Europe, NHS, state security, strikes, «killer» viruses, «gender», Strictly….. in fact anything that ruffles the punters’ feathers in powerless frustration.
    Through this media miasma the focus matter is the restoration of Scotland’s sovereignty. When even the official National party has trouble remembering its rôle sites like this are focus points.
    In this matter Scots do have the power to hand.

    Liked by 2 people

  14. Westminster voting intention (field work 2nd – 5th Dec) Savanta, multi-regression poststratification methodology. Baseline 2019 GE.
    SNP 55 out of 59 seats (+7)
    Tories reduced to one seat North of the traditional, River Trent dividing line between North and South of England.
    Non SNP seats: LibDem hold Orkney & Shetland, Lab hold Edinburgh South, Lab gain Kirkcaldy & Cowdenbeath, East Lothian.
    Quite why Labour would gain the two ALBA seats? I would put this down to mathematical misinterpretation. Sadly, the SNP would likely defenestrate Neale and Kenny. If Labour made two gains it would likely be in their historical, Lanarkshire heartlands.
    Crucially this would give a colour coded, election night, “satellite map” showing Scotland a solid yellow with a clear dividing line along the Solway / Tweed boundary. No equivocation about unionist constituencies in the Borders.
    This would be the image to present to the international community, where the true, future struggle is destined to conclude.

    Liked by 1 person

  15. remains to be seen if the leadership will ‘catch the wind’ this time or whether any momentum – generated by cost of living crises given added impetus by the UK Supreme Court ruling – will be lost due to prevarication, timidity and unpreparedness on behalf of the movement’s political leadership.

    The current NuSNP couldn’t catch a cold nor pass wind! Once again, they are going to ‘promote’ a bill at Westminster to amend the Scotland Act. Has nobody told them the Arithmetic, it won’t work!
    If, and it’s a big IF, you are serious about our Independence then it has to be via Holyrood. It is for us Scots to decide. ( I have no doubt the MSP’s won’t want to be the frontline either)

    Liked by 2 people

  16. Be interesting to see where public opinion is in the very near future as the absolutely monstrous heating bills start to land.

    For many folks they will just not be able to pay. For others they might just not pay, or substantial underpay.

    All ready the utility billers are shifting those in quarterly contracts onto monthly ‘ on demand’ contracts.

    Moreover, the utility billers are setting up their IT systems to record every aspect of their customers lives.

    From date of birth, to national insurance number, to whether you own or rent your home, whether you have a mortgage, the type of house you have, how many rooms, how mant people in the house, if folks are on benefits, the relationship of other people in the house, how you travel to work, if you have an electric car, if you have contracts with any other utilities, the right to record all conversations, right to share your information, your location when you contact them, the list goes on and on with regard to what they are going to hold. This is not an exhaustive list.

    And folks, if they find this difficult to believe, should check out, as an example OVO’s Privacy Terms.

    Clearly the obscene profit takers see energy customers as the turkeys for their plate. And in truth, thus far, they have been.

    Anyway, no doubt the next few months may change people’s, relationships with their corporate masters.

    Liked by 1 person

  17. I must admit I don’t pay a lot of attention to the polls but they do seem important to some people like John Nicolson’s rather limp attempt to claim the boost in the polls was a sign of the good job the SNP was doing as he attempted and failed to knock Kenny MacAskill off his stride in the commons. The upturn in the polls is despite the efforts of the SNP not because of it in my opinion. The speech by Kenny is well worth a look if you missed it as he was in fine form and largely ignored the intervention by John which is as it should be. Weather wise imagine how cold it would be if we did not have all this global warming to take the edge off the frost. We finally got a real person to turn up and read our meter the other day so with a bit of luck we will get our bills sorted out as the last reading I sent in I was over 600 units less than their estimate and despite my account being in credit to the tune of nearly £700 they increased my monthly DD to over £200. Chancers. In better news the council have backed down a bit on being nasty to our new Ukrainian neighbours and it looks like they will not be deported to Oban anytime soon. We have 6 Ukrainians here and after months of hard work by their sponsor and a lot of expense on his part they seem to have settled well and are busy building a garden for themselves and getting on with life. It is touching to see the kindness of many good souls offering a leg up to a family who have lost everything and restores my faith in human kindness.

    Liked by 2 people

    1. “I must admit I don’t pay a lot of attention to the polls but they do seem important to some people like John Nicolson’s rather limp attempt to claim the boost in the polls was a sign of the good job the SNP was doing”

      That is very much to the point.

      People in the SNP will point to the surveys and say “look at the polls, our approach working and aren’t we doing well!”

      And that will do for a couple of months.

      It’s superficial, and fraudulent.

      Liked by 1 person

  18. I forgot to mention a well done to Salvo and the Scottish Sovereignty Research Group for their efforts in publishing the claim of right and a lesson on Scottish history in i Scot magazine. While this may be preaching to the converted in some way it gives a hard copy that I can pass on which will come in handy.

    Liked by 1 person

  19. ‘That may have happened if the momentum so generated by events had been captured by the SNP and used to drive support with a suitably aggressive and nuanced campaign. It wasn’t.’

    Is Nicola Sturgeons attachment to the W.E.F. in any way connected to this lack of momentum and drive?

    Liked by 2 people

Comments are closed.