I wrote some weeks ago about age based demographic sub-samples in Iowa where I dropped a reference to a yet-to-be defined concept I called Expected Demographic Outcome, which I'll abbreviate EDO. The intent of EDO is to provide a structured pattern for the analysis of demographic sub-samples, I'll detail the pattern after covering some definitions.

The notion of an outcome implies some measurable result; with polling this measurable result is typically an election. With some more context this is an incredibly vague statement. Sure an election has a winner and a loser, and that's generally what everybody focuses on in any election based polling, but it also has a sample. The sample includes who voted, and the outcome, how they voted. These are the two variables, outcome and sample, we're attempting to better isolate and understand.

Generally, pollsters make assumptions about the sample while directly attempting to measure an outcome. The goal of EDO, is to normalize these assumptions so that the outcomes of multiple polls are directly comparable.

In the 2008 Iowa Democratic Caucus Entrance Poll, the gender participation percentages were 57% for females and 43% from males. I'll denote this as a +14% female gender sample. In order for a pollster, from 2008, to really predict the outcome, they would have needed to correctly guess the gender sample (among other things). Making this turnout prediction is hard and not easily gathered empirically. An incorrect prediction can obscure the actual sampled outcome. If the sampled outcome is obscured differing polls are not comparable.

In the 2016 Iowa Democratic Caucus, which occurs on February 3rd, the number of participates nor the percentage of female participants is known because the event has not yet occurred. Pollsters are making assumptions about both of these numbers (and more). It's impossible to know whether a given pollster's assumptions are good or bad, so I want a way to filter out their assumptions. This is the idea behind the Expected Demographic Outcome.

Lets take a look at the new polls from Iowa while illustrating this new idea and why it's useful. The table below includes the female participation percentage included in the sample by each pollster. It then includes the net percentage lead between Hillary Clinton and Bernie Sanders in the overall sample as well as for both male and female sub-samples. Hillary is depicted in shades of pink with a positive net and Bernie is depicted in a blueish-purple; the colors are programmatically derived from their names.

There's a lot to look at above. Lets look at the first two rows initially. In the Quinnipiac University poll ending Nov 22, Bernie trails overall by 9%. While in the YouGov poll ending Nov 19, Bernie trails by just 6. Lets make these polls comparable by normalizing the gender sample. We'll take the +19.26% Female sample from the YouGov poll and reduce it to +14% Female to match the Quinnipiac gender sample. It's a simple multiplication, and I'm neglecting the minor influence O'Malley has:

EDO = [(-13% Male) * (43% Male Sample)] +
          [(19% Female) * (57% Female Sample)]
    = -5.69%  Male    + 10.83% Female
    = 5.24%

It's not a huge change, but if the YouGov sample would have included the same distribution of males vs. females as the Quinnipiac Poll, the underlying outcome would have improved for Bernie by .75%.

Lets do the same normalization using the Public Policy Polling poll ending on Nov 1:

EDO = [(20% Male) * (43% Male Sample)] +
          [(19% Female) * (40% Female Sample)]
    = 8.60%  Male      + 20.80% Female
    = 31.4%

Again, the normalized outcome moved by less than 1% when comparing the Quinnipiac Poll to the Public Policy Polling poll. The normalizations themselves are uninteresting between the small subset of polls below because they generally all predicted similar gender samples. Now that we know the gender samples are generally comparable, we can compare their outcomes.

Of the 7 polls below, 3 of them do not provide gender specific demographic data so we can't readily normalize their outcomes and compare them. That leaves the 3 polls already mentioned and the poll sponsored by Monmouth College and KBUR. Why do two polls show Bernie winning by ~10% among men and the other two Bernie losing by ~20% among men?

A debate did occur on November 14th, so perhaps men were greatly swayed towards Bernie? By looking at pre-debate polling from the pollsters involved there is a clear pattern by each pollster that is uncorrelated with the recent debate:

YouGov and Quinnipiac seem to consistently show Bernie winning among men while Public Policy Polling uniformly shows Bernie getting crushed. So what explains this? The likely explanation is the construction of the sample. We determined the gender sample was reasonable using EDO, but there are other components to the sample we can't quantity or perhaps don't have the data to quantify.

The methodology for each of the 3 pollsters are similarly vague, but likely where the discrepancy lies.

Quinnipiac uses the following methodology to gather their sample of Likely Voters using live interviews on landlines and cellphones:

This survey uses statistical weighting procedures to account for deviations in the survey sample from known population characteristics, which helps correct for differential survey participation and random variation in samples. The overall adult sample is weighted to recent Census or American Community Survey data using a sample balancing procedure to match the demographic makeup of the population by county, gender, age, education and race. Margins of sampling error for this survey are not adjusted for design effect.

Source: Quinnipiac University [1]

Public Policy Polling contacts Usual Voters using automated phone calls and internet surveys:

Public Policy Polling surveyed 638 usual Republican primary voters and 615 usual Democratic primary voters from October 30th to November 1st. The margin of error for both parties is +/- 3.9%. 80% of participants responded via the phone, while 20% of respondents who did not have landlines conducted the survey over the internet.

Source: Public Policy Polling [2]

And YouGov uses a hybrid telephone/internet based sample over Likely Voters:

Respondents were selected for participation from available panel members to be representative of registered voters from each state in terms of age, race, and gender. A propensity score (based upon a case-control logistic regression including age, race, gender, education, born-again status, and party registration) was estimated for each respondent and responding panelists were post-stratifed upon propensity score deciles, and adjusted for differential recontact from the prior wave. A score for likelihood of voting was computed for each respondent based upon past turnout and self-reported likelihood of voting in the presidential primary.

Finally, the product of the base weights and turnout probabilities were raked to match parameters from past primary and general elections in the state drawn from exit polls, and aggregate parameters from the current voter file. The weights were trimmed to have a maximum value of seven.

Source: YouGov/CBS News [3]

It's difficult to definitively say which pollsters are using better samples, but these divergent outcomes clearly point to differing methodologies that we can't quantify given publicly available data.

Marist Polling Institute and Selzer & Co. have not conducted recent polls in Iowa.

Updated on November 29, 2015 at 6:37:42 PM CT