When Discovery Network's Learning Channel was preparing to launch Junkyard Wars, it called on audience-research firm ASI to get a read on what viewers might think of the show. The pilot was shown to test groups, who evidently liked what they saw. But when Junkyard Wars debuted the night of July 5, it landed in the midst of a network battle, and the only survivor was CBS' Big Brother.
The test audience "could have loved [Junkyard Wars], but compared to what?" asks Discovery's Steve McGowan, senior vice president of networks research. "It's tough to use this research as an absolute."
Granted, but networks, both cable and broadcast, stake their schedules on the predictions of media researchers. They test everything from pilots to promos, as well as tweak existing programs-storyline changes and new characters-and provide a multitude of marketing-research services.
The field of audience research is dominated by a handful of players such as ASI Entertainment, Frank N. Magid Associates and TNS Intersearch. But there are smaller firms in the arena with notable clients. CBS, for example, turns over its pilots to Robert A. Brilliant Inc., which calls itself a "boutique" business. A Canadian company, Gadd International, has done a project for Discovery and is "hustling" to sign up other U.S. clients.
Nielsen Media Research, which dominates the TV ratings business, is expanding into pilot testing. Nielsen's Paul Lindstrom, vice president, custom research sales and marketing, says the company is also "doing research on programs that are already on the air, experimenting quite a bit with the use of the metered sample," he says. "We've been examining minute-by-minute losses to evaluate when viewers switch away from programs."
Strategic Media Research (SMR), a 20-year-old research and marketing company that has specialized in radio, is now testing TV promos online. Clients include MTV, VH1, Comedy Central, Country Music Television and The Nashville Network.
Multiple methodologies
Testing methods vary, and each company claims its approach is superior.
ASI, headquartered in Glendale, Calif., says that, in the 1960s, it was using theater-based research using dials, questionnaires and focus groups. In Los Angeles, ASI has three theaters, including Preview House in Hollywood, equipped with dials that viewers use to convey their reactions as they watch different programs. The system, also called dial-testing, includes a tune-out button, which ASI says is an important new feature exclusive to ASI testing that registers that dreaded moment.
The company also says it was an early user of in-home cable testing: Viewers tune into a designated cable channel to watch a program and are questioned by phone about their reactions.
"There's no research methodology that we have not experimented with," says Paul Lenburg, executive vice president of the company. ASI is co-owned with Audience Research & Development of Dallas, a news research and consulting firm, by parent company Media Advisors International.
Cable testing is the preferred approach of ABC network, although it relies on other methods such as focus groups and phone studies. TNS Intersearch does the lion's share of pilot testing using cable for the network.
"The key to cable testing is that [viewers] are at home. They're in a natural environment watching a program. We use a number of cable systems to get as strong a sample as possible. It's a tried-and-true technique," says Edye Twer, senior vice president, media and entertainment for TNS, headquartered in Horsham, Pa.
NBC uses the cable-test method for pilots as well, and enlists ASI and TNS for the work. "These episodes all come in at the last minute, and it allows us to test as many as possible in a very quick time frame," says Mitch Metcalf, NBC senior vice president, program research.
Magid Associates, which uses a variety of methods-including dial-testing-for pilots and promos, is no fan of cable testing. Dave Smith, president of the company's entertainment and new-media divisions, feels there are drawbacks.
"You can only test cable households, and cable companies have fewer and fewer available channels to do that sort of thing," says Smith. "Plus, not all cable systems that participate in testing have equal ethnic representation. You can't get a true national sample."
Another possible drawback with in-home cable testing, says Robert Brilliant, is that you have very little information about the viewing conditions.
Magid, a 42-year-old company known for local TV news research and consulting, swears by its "proprietary" methodology, in-home VCR testing. Test viewers receive videotapes of the programs and are later questioned by telephone. "We can talk to anybody with a telephone and a VCR. And VCR's are at 99% penetration," Smith says.
The firm, based in Marion, Iowa, is expanding into network TV programming consulting and boasts that it has tested 22 pilots for FOX this season. Brent Magid, president, says a new method was used, but would not describe it except to say that it is "certainly an innovation beyond what we were using a year ago."
Brilliant's company, based in Sherman Oaks, Calif., tests in Las Vegas by setting up rooms near high-traffic areas and intercepting people as they walk by. Brilliant says that every program is tested the same way ensuring that "any differences in scores can be attributed to the program."
Toronto's Gadd, whose main client is the Canadian Broadcasting Corp., uses computer interviews to test TV programs and promos. The audience sample is recruited by phone, and the company delivers a videocassette of the program. After watching the tape at home, they go into Gadd's facility and complete a computer interview.
"We like to get overall impressions, and we ask a lot of open-ended questions-likes and dislikes, thoughts and feelings-so people can express in their own way how they feel about a show," says Mike Gadd, who founded the company 11 years ago.
Internet access to audiences
As ASI's Lenburg points out, TV research methodology has been evolving for decades. "In time, the systems we're using today will slowly but surely ebb as new systems come on stream," he says. The Internet, inevitably, is beginning to join other tools of the trade.
SMR, headquartered in Chicago, is already streaming audio and video clips for its clients, which include cable networks. Feedback from participants is instantaneous, says Deborah Richman, president of SMR. "As soon as respondents complete the survey, the client can get the results, and it's completely analyzed in charts. Therefore, the cable network can adjust very quickly to what it learns."
Richman says respondents are selected by recruiting from the client's Web site, by purchasing e-mail lists, through telephone recruitment or e-mails to groups that have responded to a contest or marketing activity.
Intersurvey, a 2-year-old company in Menlo Park, Calif., uses audio and video streaming to test TV commercials, promos and program concepts. Respondents, recruited by telephone for its Web-based surveys, are given customized WebTV as well as Internet access and e-mail at no charge. Each individual completes one survey a week. The company has pre-recruited 80,000 consumers from which it draws various sample sizes.
"A unique advantage that InterSurvey offers in TV advertising or any material targeting a mass audience is that our panel is selected to represent the U.S. population but includes consumers who would not previously have had Internet access," says Jean Durall, director of marketing programs for InterSurvey.
And although ASI would not provide details, company President William Taylor says the company is "within a few weeks" of announcing the launch of a "new methodology" that will embrace all the new technology.
No one disputes that testing pilots and other programming using video streaming is on its way. But it's not here yet.
"You want good quality, not the jerkiness that you get with broadband," says Brilliant, when it comes to entertainment research. "Getting a sample that represents the television audience is difficult, since Internet penetration is only around 50%," points out CBS' Ted Kneisler, director of primary research.
But when the kinks are worked out and penetration grows, evaluating programs via the Internet will "give us instantaneous feedback," says Discovery's McGowan.
It takes guts
No matter how audience research is collected and analyzed, evaluating TV programs is an inexact science. As Brilliant puts it, "a series is a moving target. You can have programs that do well one time, but that may be hard to replicate."
And TNS' Twer points out there are many variables: the time of year, the time of day a show airs, the competition and the episode itself.
But there are a few simple tools that ASI's Lenburg thinks often are overlooked in the quest for the next Survivor or Who Wants to Be a Millionaire?
"There is no substitute for creativity, experience and just plain old guts. What we do is add a little objective information that connects the creative people, the distribution people and the studios with the consumer. Our business is connectivity."
