Viral Polls Are Not Representative Polls

(kevinreuning.com)

36 points | by m-hodges 4 hours ago ago

57 comments

  • swipeharbor 4 hours ago ago

    This filter isn't specific to polls. Anything that reaches people through reshares gets selected on how surprising or pleasing it is, so the visible sample skews toward the tails. Marketing has the same problem with screenshots of campaign results: the ones that travel are the outliers, and anyone calibrating on what they've seen shared ends up with expectations nobody can hit.

    Curious whether the effect survives controlling for pollster. Some pollsters have consistent house effects and some have much bigger followings, so part of the slope could be "polls from well-known outfits get reposted more" rather than "favorable margins get reposted more".

  • __MatrixMan__ 3 hours ago ago

    I think we should do our best to ruin polling data.

    Polls are the first step in a pipeline that effectively turns money into votes. The better that pipeline functions, the worse our democracy functions. Let the only accurate poll be the election itself.

    • acgourley an hour ago ago

      I believe you're saying "Information is power and it can be used by an adversary to the people to outmaneuver them."

      True, but that's not the only party which can use this. The people themselves can use it to build better coalitions and to hold representatives accountable for acting against their will.

      • __MatrixMan__ 41 minutes ago ago

        I'm all for better coalition building, but is it truly the same data that enables both?

        Expecting people to share their political leanings with any stranger who asks without encouraging them to make a friend/foe determination beforehand seems like it opens the playing field to interference just as well as it does to good faith feedback. Surely we can find a way to bias this interaction in the right direction somehow.

        • acgourley 11 minutes ago ago

          It's a good question. I prefer longer term questions about more fundamental beliefs, as that's what coalitions can form around. It also opens the door to solutions which take longer (more likely to work). Specifically, as an American, I advocate for a robust polling regime well before the primaries to encourage more popular candidates and slates to emerge.

    • openasocket 2 hours ago ago

      I actually really disagree with this. If we want our representatives to reflect the will of the people, they need to have an accurate understanding of their electorate. When issues come up we need a way for the people to be heard, and polling is an important part of that.

      • acgourley an hour ago ago

        This conditional is true, but there is a related one: If we want our representatives to outmaneuver the electorate on behalf of donors, they must have an accurate understanding of what their electorate is paying attention to, and what they are not.

        Surfacing understanding of the electorates beliefs and desires is good but double edged. I think we can balance it out with yet other strategies and don't agree with GP's cynicism here but it's worth taking on directly.

      • __MatrixMan__ 2 hours ago ago

        Isn't there a pretty clear line between the following?

        > I'll give you a gift card if you tell me how you're going to vote tomorrow

        and:

        > I want to represent you well, lets talk about what's important to you

        I wouldn't propose we interfere with the latter. Or is the former truly useful for improving the quality of representation? It seems to me like it's really only good for improving ad targeting. Any representative worth their salt can find a way to get feedback from their constituency without requiring the people to participate in a cold call from a research company whose objectives are unknown.

    • saulpw 3 hours ago ago

      How do you confirm the election results itself though, when the election purveyors can't be trusted? Statistical polling has been a third-party check on elections for a century.

      • giantg2 2 hours ago ago

        They don't really check anything, do they? When there is a discrepancy between official results and polling, the general response was that the polling missed something, not that the official results were wrong.

        • yorwba 2 hours ago ago

          You're probably thinking about pre-election polls. Those are frequently off by a few percentage points even if they took great care to take a large representative sample of the population because it's difficult to predict turnout. However, there's one opportunity to get a representative sample of voters: by conducting an exit poll right after they voted. If exit polls show a large discrepancy from actual results, it's usually seen as an indicator of fraud.

      • __MatrixMan__ 2 hours ago ago

        Well that seems pretty shaky. You're saying that in order to cast doubt on an election, I don't need to actually target the election itself? I can just generate a bunch of disinformation in an unregulated, less-scrutinized, adjacent channel?

        To answer your question, you volunteer locally so that you're involved in the process, or you encourage somebody that you trust to do so, or somebody they trust, etc... You validate it by doing it validly.

        • saulpw 2 hours ago ago

          There are many polls vs one election. It's a lot harder to manipulate all the polls.

          Also, yes, we can and should be volunteering locally. AND ALSO we can have independent statistical review.

      • tyromaniac 2 hours ago ago

        Not with these margins

    • nativespecies 2 hours ago ago

      I love people on the internet who are confidently wrong. It's impressive

      • __MatrixMan__ 2 hours ago ago

        Am I missing something? If so, please talk me out of it.

        It's just clearer communication to state your position plainly rather than bedazzling it with a bunch of disclaimers.

    • tuveson 3 hours ago ago

      > Polls are the first step in a pipeline that effectively turns money into votes

      What? You’re stating this like it’s self-evident, but I don’t follow.

      • __MatrixMan__ 2 hours ago ago

        Well if you want to spend money on ensuring that a certain candidate wins, probably you want to spend barely enough to make it happen, right? Like, if your alternatives are to spend a few billion in an untargeted way, or spend a few thousand in a targeted way, then you're going to want to chose the latter for a variety of reasons.

        Wouldn't polling data be the place you'd start when it comes to making that decision?

        • giantg2 2 hours ago ago

          Isn't this the main concept of the Cambridge Analytica story - targeting people in the swing areas, and even based on personal information?

          We're already here and I think personal informantion, behaviors, etc collected from social media and corporations would be the bigger threat than the polling itself.

          • __MatrixMan__ 2 hours ago ago

            Yeah, we should probably be trying to jam those channels as well. Surely we can find a way to identify and display messages client-side from the people we want to hear from, while filtering out noise from any decoys, such that the platforms can't figure out who is a decoy and who is a human, rendering their product worthless to a would be election interferer.

            But that's a whole endeavor. Ignoring or lying when some research company wants to know about your political beliefs is a nice easy start.

        • tuveson 41 minutes ago ago

          I guess? But non-monied groups (e.g. volunteer doorknockers, phonebanks) would use the same data to decide where to focus.

          Something about trying to deliberately distort the public about the likely status of an election also just feels icky to me. Likely to cause more harm than good.

  • bombcar 4 hours ago ago

    Almost all polls are worthless, because the decision to respond to the poll is itself a major portion of the signal.

    It takes extensive work to get around that problem alone, let alone "viral" and other worthless push-polls - work that even the very best in the industry, at the highest level, can make major mistakes on.

    • gruez 4 hours ago ago

      >Almost all polls are worthless [...] because the decision to respond to the poll is itself a major portion of the signal.

      But most polls I see are from professional pollsters like Pew or YouGov? For them, correcting for this is opinion polls 101. That's why you see "representative sample". They're not just randomly polling 1000 people and taking the average. They're adjusting for a bunch of other demographic factors as well.

      • asdff 3 hours ago ago

        No matter how they adjust demographic factors they cannot control for the willingness to respond to polls at all. They are always subsetting this weird population set that shares that strange variable of being willing to spend ten plus uncompensated minutes with a spam call pollster.

        • mjdv 3 hours ago ago

          > they cannot control for the willingness to respond to polls at all

          Why not? If they poll the same question (e.g. political party) repeatedly (e.g. in many elections) eventually you could build some kind of model of which parties poll-answerers are more likely to skew towards, or how much they shift compared to non-answerers. Not fundamentally a different process from other ways of removing bias from polls.

          • bombcar 2 hours ago ago

            That’s exactly what they do but by definition it will always be a lagging indicator when the game changes.

        • qrobit 3 hours ago ago

          > [...] they cannot control for the willingness to respond to polls at all.

          I believe this is not true, because at least ESS (European Social Survey) corrects for non-response, although how exactly is beyond my understanding of survey analysis. Each country reports the response rate as well. See [^1] section 3. Although ESS is much larger survey than simple opinion poll, there are political affiliation questions too (`^prtv.*$` variables).

          [^1]: https://www.europeansocialsurvey.org/sites/default/files/202...

          • asdff 2 hours ago ago

            Appeal to authority here. If you know statistics you know the truth is basically impossible to model. The great lie of statistics is you are measuring associations that are true in your dataset and you are making assumptions that this dataset will be more or less representative of the population and perhaps sampling 10 of 1000 will be sufficient depth to appreciate this true result.

            Sometimes this works out alright. But sometimes not. Too many factors in the universe to model. Never enough power to look at all them even if you knew what they all were. It works well when a few dominant factors are present but when its a million latent factors with technically every pairwise interacting relationship present in some degree, good luck.

            • bombcar 2 hours ago ago

              Polls are either accurate or not.

              Professional polls are either accurate or have a good reason afterwards why they weren’t.

              • asdff an hour ago ago

                Exactly, and therefore they must be considered as such. Models based on the finite data that were collected and handful of conveniently observable variables selected under study, nothing more nothing less. Whether you decide to use this as evidence for reality is at your own peril. Not just polls but truly any question involving statistics.

      • toomim 4 hours ago ago

        And there's no clear way to adjust for the most basic factor that people who vote for one candidate are less likely to say so.

        In order to account for that, you need you know how many people who are less likely to respond are voting for that candidate. And then you're back to square one.

        This is why professional polls have had a consistent bias over the last few years, as this hidden unobservable factor's effect has gotten stronger and stronger.

        • bombcar 4 hours ago ago

          And even the best pollsters, who over decades learn to correct for drift like that, get blown up by a sea-change that they didn't expect.

      • IcyWindows 4 hours ago ago

        I don't see how they can adjust for the type of people not responding to polls unless they can force them to answer.

        • bombcar 3 hours ago ago

          Because they can correct against the election results.

          If you're trying to do a scientific poll that is not about an election, it can become exponentially harder unless you can find independent ways to obtain data that helps you adjust (like home ownership, which is reported to the government).

          • asdff 3 hours ago ago

            Retroactively but thats not why anyone looks at polls. By that point you’d throw out the poll and deal with the real result. And really theres no shot whatever corrective factor remains the same for every election and poll relationship. There are too few elections over too long time over a constantly shifting electorate to generate enough power to appropriately qualify this divergence.

        • gruez 4 hours ago ago
      • bombcar 4 hours ago ago

        Then you're mostly looking at those who "try" - but if you look wider you'll see polls used way more often than that (think of the companies entirely built around it, customer satisfaction surveys, Facebook "polls", etc, etc).

        Doing a proper poll is expensive and (frankly) most of the time is pretty predictable or you can get 90% of the results with 10% of the effort, and most times you don't actually care about the results because you're building it to get a result you want anyway.

        (Interestingly enough elections themselves almost by definition have notable discrepancies to a "true" perfect sampling; but it's usually overcome by the sheer volume.)

    • rurp 3 hours ago ago

      This is totally wrong. Accounting for partisan non-response bias is a standard part of a pollsters job, not some insurmountable gotcha. Some are better than others at this. Polls aren't oracular, nor are they claiming to be. It's almost always better to look at an average of polls rather than a single one.

      There is a lot of public data on past elections that anyone can analyze and see exactly how accurate polls have been in the past. If you're expecting polls to be a perfectly accurate prediction then you are misunderstanding their purpose. What they are great at is giving a probabilistic estimate of the outcome, within a certain range.

      It's likely that any key race you care about next week will end up within 4 points of the current polling average. It's extremely likely that it will end up within 8 points. That's useful information!

    • bluegatty 3 hours ago ago

      "because the decision to respond to the poll is itself a major portion of the signa"

      Yes - they are aware of that.

      • asdff 3 hours ago ago

        And how do they correct for that? What assumptions are made in that process?

        • bluegatty 2 hours ago ago

          They do a variety of things based on professionalism, budget, ability to gauge that.

          My point is --> this problem is the most basic and fundamental problem that is the focus of every pollsters existence <--

          Asking questions is extremely easy.

          Getting a 'truly random sample' is extremely hard.

          Trying to factor in those things is 100% of the work, that's what they do, it's their job.

          Some are better, some are lazier, some have better data, some are individuals, some are teams so it varies hugely.

          When the conditions are controlled they'll have a better idea than otherwise.

          Their job is not 'asking questions' it's 'figuring out how to get random samples'.

          • asdff an hour ago ago

            This is appealing to authority, assuming the work ought to be perfect just because people are professionals. What if there is no good way to control for these latent factors? What if the latent factors change by the year and their magnitude of effect on the overall result changes? All this happens. No matter what you model with statistics, you are only ever presenting your model and its assumptions based on the data that were used with it. You can never say "now I understand reality" if you actually believe in statistics as they are and not just the popular religion of 'statistics'.

    • sublinear 4 hours ago ago

      > Almost all polls are worthless

      Do you have one example of a poll that isn't?

      • asdff 3 hours ago ago

        A poll you serve to people you know who will respond and respond in good faith accurate to their opinions. E.g. polling for availability to go to a baseball game with friends. Beyond that, well, here be dragons.

        • sublinear 17 minutes ago ago

          Tell me you haven't planned a wedding before.

      • Legend2440 4 hours ago ago

        The only poll that matters is the one on election day.

    • wtch98 4 hours ago ago

      Most professional pollers do a good job.

      When they are 3 or 4 percentage points off they are ripped to shreds.

      The polls for the recent Brazil elections were about 49% for Lula, the result was 52%, that's a 3 point difference, not a massive error margin. With brexit the polling pointed to about 52% for remain, actual result was 48%.

      But those are proper polling companies, not worthless online ones.

  • bluegatty 3 hours ago ago

    The very nature of the term 'viral' obviously would be disqualifying or hint that something is very awry.

    The entire objective of a polls is to get a probabilistic sample. That's the whole thing. Anything that deviates that ... creates problematic outcome.

  • dlcarrier 4 hours ago ago

    Of course polls that show web unexpected lead for an individuals favorite candidate are more likely to be shared by that individual.

  • tonymet 3 hours ago ago

    All polls are not representative. Stated preferences are not revealed preferences. People are terrible at answering honestly. You have to surveil their behavior to understand them.

  • yipinwong 4 hours ago ago

    All polls are flawed. Remember last US pres election polls by CNN? Completely wrong. Why? Because they surveyed mostly left-"learing" participants.

    Right and supposedly middle (this doesn't exist) media does the same.

    That means we gotta find the average as you suggested.

    ---

    Side note

    > Given that Musk has absolutely destroyed the Twitter API for academic use

    Why bring up Musk, when you can simply say, "Twitter lowered Academic API rate limit because ...". You sound like you need someone to blame not professional and politically motivated (yes the post is about the political poll but this shows the "bias" and his supposed poll will be biased depending on how he gathered data).

    Twitter would have the same way of generating metrics for the justification as you are trying to justify the flaw in the poll outcome.

    Most likely the decision is based on their internal metrics/dashboard report

    • bluegatty 3 hours ago ago

      These are all problematic statements.

      - CNN doesn't do polling, they have firms do that

      - All reputable firms try get valid results, they have professionals, they know who they are talking to and the limits. What we see in the last 2-3 weeks of the elections are the polls they want to release however, those that err in one direction or another.

      - They definitely accommodate for misrepresentation samples - if more of their respondents indicate 'registered republicans' than in the general public they will adjust.

      - It's very hard to measure 1) the people that are hard to reach and statistically more likely to not trust pollsters and 2) likelihood of actual voting aka enthusiasm.

      - there are plenty of centrist institutions and pollsters

      - It's fair to call out Elon because he's making the policies

      Twitter is just a bad way to measure anything except for the Zeitgeist of Twitter readers.

      • yipinwong 3 hours ago ago

        I am upvoting your comment for correcting my knowledge on CNN not doing their own polling.

        Some I do not really agree, as reputable firms != trustworthy polls.

        e.g. Check out Times magainzes "Best Inventions of the Year, John Termus" (https://time.com/collection/best-inventions/2026/apple-iphon...) wtf is that. Time magazine might be "reputable" but whatever they discuss is not trustworthy given Appl duo being crap.

        • bluegatty 2 hours ago ago

          TIME magazine is a tabloid-ish reader publication. It's popular editorial, not 'highly objective news'. Who TIME puts on their cover has nothing to do with the quality of the polls they might cite.

    • kijashdkujdfhas 3 hours ago ago

      In 2021 Twitter was encouraging free academic use of the API [0].

      This was in line with long standing behaviors in this space [1].

      Musk bought Twitter in 2022. In 2023, they began substantially restricting API access versus how it had been for years.

      It is not unfair at all to look at that timeline, look at Musk's acquisition as the biggest change that had happened to the company in that timeframe, and draw a relationship.

      [0] https://newsonline.library.vanderbilt.edu/2021/03/use-twitte...

      [1] https://arxiv.org/abs/2404.07340

    • jasonlotito 3 hours ago ago

      > Remember last US pres election polls by CNN? Completely wrong.

      I do. I don't remember that at all. I think you might have some odd definition of completely wrong though.

      But please, by all means, share these 100% wrong polls that showed, for example, Trump having 0% chance of winning. Most of the polls I saw were fairly accurate within their margin of error and made it clear the race was tight (which, in the end, it was). And nothing I've seen has changed that, but if you have polls from CNN specifically that you can point to, by all means.