Change Research poll with David Jolly ahead of Byron Donalds warrants heavy salting

A new poll from Change Research shows David Jolly leading Byron Donalds by 5 points, 47% to 42% — and among (self-described) likely voters, Jolly is up by 6 points.

Maybe it’s time to buy stock in salt?

We are going to start with the positive aspects of this poll.

The methodology is perfectly transparent, so kudos. They disclose the client (Freedom Project USA), the recruitment sources, the weighting variables and the post-stratification targets, and they correctly distinguish between a true random sample and what they actually did.

Citing Pew’s modeled margin of error convention and Kish‘s formula is a sign of statistical honesty — they are not pretending this is something it isn’t.

The sample size of 1,273 respondents is reasonable for a statewide Florida survey, weighting on six variables (age, gender, race/ethnicity, education, 2024 presidential vote and region) against the actual voter file is methodologically solid, and the party balance appears consistent with what many pollsters currently believe will happen this November (+12 GOP) …

But …

Why are we even talking to unlikely voters?

Why is there no uninformed test ballot?

And why was a large portion of the sample self-selected and self-described — even to the point of including people who say they are not likely to vote (20% of them say so, so why are we even talking to them)?

So here we go:

— Opt-in, self-selected sample: At least part of the sample is not a probability sample. Recruiting through Facebook/Instagram ads systematically misses people who aren’t on those platforms or don’t engage with political ads — older, less online, lower-engagement voters. The text-message channel helps but doesn’t fully solve this, and all the weighting in the world cannot fully correct for who chose to respond and who didn’t. And keep in mind that today Democratic enthusiasm is very high, and as a result likely-voting Democrats are more likely to want to show their anger/enthusiasm by participating in these kinds of polls. In short, self-selection (also known as “opting in“) can and often will bias a poll.

— The client: Freedom Project USA commissioned this poll. That’s not disqualifying, but commissioned polls should always get more scrutiny. And the pollster is a left-leaning one (heck, Kamala Harris is on their home page!).

— English only: Florida has a substantial Spanish-dominant population, particularly among Puerto Rican and Cuban American voters. Conducting in English only could meaningfully skew results among Hispanic subgroups.

— No straight ballot test: We truly cannot understand why a poll testing where the race stands today would not include a straight, uninformed test ballot. Instead, they use an “informed” test ballot, meaning that instead of simply asking who voters would support, the pollster first reads a descriptor of the candidates. Here is their rationale: “Given that neither David Jolly nor Byron Donalds are well known to Florida voters, the survey asked voters to consider the following short messages and contained a photo of both candidates.” Wait — what?! So the ballot question here is based on a narrative (and a photo), neither of which will be on the ballot come Election Day. Having seen this game played out too many times, these narratives often contain hidden (or not-so-hidden) biases. For example, here Jolly is described with 84 words while Donalds gets only 67. Yes, this matters! The simple weight of a message containing more positive statements about one candidate versus the other does make a difference. That’s an objective measure (word count), but when we also take a subjective look, the descriptors seem to favor (in my opinion) Jolly. Jolly’s descriptor is more about the top-polling issue of affordability, while Donalds’ leans on endorsements, “corporate America, the financial services industry,” etc. Further, the Jolly descriptors are framed as universally positive messages (work, wages, health), while the Donalds descriptors are framed in combative, divisive language. But the most important question recurs: why not ask the straight ballot language to see where this race actually stands?

— Self-described voter behavior: As repeatedly noted in past columns, asking someone whether they intend to vote — versus basing voter probability on past voter behavior — also tends to bias results, as those enthusiasts will of course claim they are going to vote. It’s good to know and can be used in targeting for a campaign, but if these answers were accurate, we wouldn’t see only 60%–70% of voters showing up in Midterm Elections; it would be more like 90%.

— Self-described voter registration: AGAIN, as repeatedly noted in past columns, asking someone how they are registered to vote — versus tracking it from the voter file, where we know how someone is actually registered — also tends to bias results. To use a specific example: today it is no surprise that Donald Trump is very unpopular with large swaths of voters. It is more than conceivable that a centrist Republican might not want to declare that he or she still belongs to the party, even though that person is still registered as a Republican.

— Verdict: Bring on the Morton’s: The transparency and methodology notes are genuinely above average for this type of poll, so it is not junk. But in our opinion, the opt-in recruitment, single-language fielding, and use of a narrative (and a photo) for a test ballot — as opposed to simply asking the question — mean you should take these findings with a whole lotta salt.