TAICHUNG (Taiwan News) — Political analysts and commentators rely heavily on polling data. It is useful in capturing a moment in time to understand what the public supports or the state of a candidate’s campaign.
Not all polls are created equal, however. Some are more professional and reliable than others, and some may have partisan leanings or intent.
Different polls produce different results. For example, in most polls, the Taiwan People’s Party (TPP) presidential candidate Ko Wen-je (???) currently leads the Kuomintang’s Hou Yu-ih (???), but some polls show Hou leading Ko.
That polls produce conflicting results is not necessarily a problem of poor methodology or political bias. Consider if three seperate polls asked the following questions,In three months, who do you intend to vote for?, If the election were held tomorrow, who would you vote for?, Who do you support to be the next president? These questions may produce slightly different results, especially in the “I don’t know” category.
Different polling outlets use different methods to collect their data, and may weight their sampled data differently. Some polling outfits call landlines, others cell phones, some use both, while others use digital methods, like emails, texts and so forth.
These differences in method mean pollsters will reach somewhat different respondents. The polling firm then tries to adjust for this and for irregular numbers of respondents in various demographics using their own formulas.
With all the diversity of polling methodology and the end results, it can get a bit confusing as to where the race actually stands. One common way of analyzing these polling variables, which also produces a simple snapshot of the race is a weighted “poll of polls.”
Chart by the author
Nathan Batto led the way
After compiling data since early September, the Taiwan News Poll of Polls is ready to be released. This is not the first “poll of polls” in Taiwan; that honor goes to the Frozen Garlic Poll of Polls by Nathan Batto, an Associate Research Fellow at the Institute of Political Science, Academia Sinica and Jointly Appointed Associate Research Fellow at the Election Study Center of National Chengchi University.
Batto’s work is superb and his Frozen Garlic blog is an absolute must-read for anyone following Taiwan politics. It has been a significant influence on my work. Unfortunately, due to personal reasons, he has not been doing a poll of polls this election cycle. Generously, he looked over my poll weightings and he gave some tips, suggestions, and insights, all of which were incorporated into the Taiwan News Poll of Polls (TNPP).
The TNPP is a rolling, weighted poll of polls. The basic poll of polls shows results from polls released in the previous 15 days, while the line chart version shows those results released every five days to provide a snapshot of how the race has evolved over time.
Fifteen days allows for enough polls to be included so that the results aren’t dominated by only a handful of polls that might skew the results. In practice, because polls take time to conduct and prepare, this means the results range from around 19 or 20 days prior to about two days prior to the TNPP’s update.
Certain types of polls are not included in the TNPP. Internal party polling, when parties release such results to the public, are not included. Also not included are internet polls of a particular media outlet’s readership because their results are biased and unadjusted.
Chart by the author
Weighting polls
Of the 19 outlets releasing polls counted in the TNPP, the majority have an average weighting, while six are weighted lower and five are weighted higher than average. There are various factors that go into determining if a poll should be given increased or decreased weight, including track record, consistency of results with other polling firms, and between polls by the same organization, the reputation of the outlet or the pollster they employ, transparency and methodology.
Polls that are consistently out of sync with other polls are down-weighted. TVBS, which used to be highly reputable for their polling, has in recent years consistently released polls that are significant outliers from all the others.
Another factor is consistency. Taiwan Public Opinion Foundation (TPOF) polling suggests the public is prone to erratic and wild mood swings from poll to poll even in the absence of any dramatic political events. By contrast, Formosa polling shows a far smoother evolution of public opinion from poll to poll, with bigger changes in the results consistent with events unfolding in the press.
One higher-weighted outlet that might surprise some readers is Want Want China Times, in spite of their deep, deep blue editorial stance. Their track record has been decent and originally I had them as “average,” but Nathan Batto pointed out that the pollster they use has an excellent reputation.
The partisan leanings of the outlets releasing the polls were not considered as a factor in the TNPP. For example, pan-blue United Daily News (UDN), Broadcasting Corp of China (BCC) and pan-green SET News (SETN) are all weighted the same. I’ve seen no evidence of these outlets colluding with their pollsters to influence the results, and frequently they’ve all released polls that do not paint a picture that their side necessarily wants to see.
Most of the lower-weighted outlets are fairly new and without strong reputations that use digital-only methods. One exception is RWNews, whose results during last year’s local elections were not bad, at least good enough to keep them in the average weighting category.
Transparency matters
Transparency is another factor to consider. Outlets that release all their data broken down into demographics, and those that provide more details on their methodology are given more consideration than those that do not.
For example, Formosa gives detailed breakdowns of their data, which is not only transparent, but also very helpful in doing analysis. TPOF by contrast provides little or none, often not even the dates when the polling was done.
Worryingly, the director general of the National Security Bureau (NSB) Tsai Ming-yen (???) said that he thinks China is working with survey research companies to manipulate opinion polls. There have also been allegations that pollsters can be bribed.
I have not seen any proof of either, at least not obviously, though it is possible there is some padding going on with a couple of percentage points here or there. I do not think the more established media outlets, with one possible exception (and they are underweight as a result), are doing this. Most polling firms would not engage in such behavior because the reputational damage would be too high if they were caught.
One of the functions of having a weighted poll is to help mitigate against this. Most of the newer, online-only news outlets that might be desperate enough to engage in this are weighted lower, as are outlets that consistently produce polling that is inconsistent with the others.
Polling is never perfect
By its very nature, polling can not be perfect. At the very least, it is a few days old.
The goal of a poll of polls is to try and smooth out the irregularities from poll to poll. Still, it won’t be perfect, but the goal is to produce something that is fairly representative of the state of the election.
One thing polling can not do is accurately predict the exact results on election day. Polls can only indicate trends. In large part, this is because so many people respond with”I don’t know/undecided”during polling, and only make up their minds on election day.
Trying to determine how those voters will vote in the end is the work of analysts, polling is merely a tool in their toolkit.
