Poll standards

The leading organization representing the nation's pollsters criticized CBS News and the New York Times on Friday for releasing results of a nationwide poll the survey-researchers organization said was conducted using an unproven methodology.

Last Sunday, the two news organizations unveiled their "Battleground Tracker," an online survey updated each month. The poll -- conducted by Internet pollster YouGov -- interviewed more than 100,000 people nationwide, the news organizations said, allowing them to project results for each Senate race in the country.

The results were featured on CBS's "Face the Nation," and the New York Times "Upshot" data-driven vertical published multiple entries about the survey. The first edition of the "Battleground Tracker" showed Republicans leading in 51 Senate races, which prompted The Upshot to project that Republicans had a 60 percent chance of winning control of the Senate.

The American Association for Public Opinion Research's statement on Friday criticized CBS and the Times for using a survey method that has "little grounding in theory" and for a lack of transparency.

"[M]any of the details required to honestly assess the methodology remain undisclosed," according to the AAPOR statement, issued under the organization's letterhead and signed by president Michael Link. "This may be an isolated incident with the Times / CBS News providing more information on this effort in the coming weeks. If not, it is a disappointing precedent being set by two of our leading media institutions."

CBS News elections director Anthony Salvanto disputed that charge, saying in a statement that CBS disclosed the methodology for the study "in great detail."

"Battleground Tracker is a pioneering project that delivers a comprehensive look at the electorate for viewers and readers in a manner that has never been seen in midterms before," said Salvanto. "As always, the methodology is available in great detail along with our findings at cbsnews.com. The Battleground Tracker model represents another example of the rich history of industry-wide innovations from CBS News."

The two news organizations issued a joint statement defending the survey.

"The New York Times and CBS News are proud to continue our long history of presenting new and methodologically rigorous analyses to our readers and viewers," the statement says. "Our findings and methods for the new Battleground Tracker were laid out in great detail for all to see, and the underlying methodology is already well-known to, and widely used in, the survey research and scholarly communities. Moreover, both The New York Times and CBS News have successfully used online panel data in past projects. We remain committed to, and on the forefront of, the highest standards of transparency and data collection[.] Battleground Tracker provides yet another example of both."

AAPOR also hit the Times individually for abandoning their standards to report results of the YouGov study. In the past, Times reporters were prohibited by an internal policy from reporting results of polls conducted among web respondents who were not randomly selected to participate in the survey.

An update to the Times' polling standards on Monday said: "The world of polling is currently in the midst of significant change, and The Times has begun a process to review its polling standards. While the process is ongoing, the paper will be making individual decisions about which polls meet Times standards and specifically how they should be used. As technology changes, we expect there will be multiple methods for capturing public opinion; we also fully expect that there will continue to be a proliferation of polls that do not meet our standards."

But to Link, the AAPOR president, that sounded like a complete evisceration of the Times' standards.

"This means no standards are currently in place," Link wrote. "It is unclear why the decision was made to pull the existing standards before the new ones are developed, vetted and published. Yes, all responsible institutions need to review their standards periodically, making appropriate changes as technologies and methodologies change. However, standards need to be in place at all times precisely to avoid the 'we know it when we see it (or worse yet, 'prefer it')' approach, which often gives expediency and flash far greater weight than confidence and veracity."

New York Times Upshot editor David Leonhardt did not respond to an email seeking comment. Earlier this week, he addressed similar criticisms of the YouGov methodology in an email conversation with POLITICO -- citing YouGov's similar project in 2012 with CBS News, a project with which the Times wasn't involved but studied closely.

"We remain skeptical about many online polls, but YouGov is clearing a high bar in terms of transparency. That's crucial for survey work, especially a newer form," he said.

(POLITICO commissions online polling using probability-based sampling, not the opt-in panels that YouGov used for this study.)