Media Metrix measures Web sites' traffic by installing software on volunteers' computers (about 40,000 today), tracking their usage and extrapolating the results -- just as the Nielsen TV ratings do. The company blankets the Net with its survey results. Among industry insiders, ad buyers and journalists, it has become the de facto yardstick for who's up and who's down on the Web. And now it has acquired the ability to "move the market" -- the ultimate symbol of power in financial circles.

This is pretty heady stuff for a company that admits its statistics are incomplete -- they're missing significant numbers of users in the workplace and outside of the U.S.

"As much as 40 percent of a typical Web site's traffic emanates from outside the U.S.," according to a white paper authored by Bob Ivins, senior vice president of Media Metrix. The report goes on to explain that the company only measures U.S. Web traffic.

At the same time, its panel of 40,000 Web users includes only 7,000 who surf from work -- skewing the results heavily in favor of home-surfing behavior, despite estimates that workplace surfing accounts for as much as half of all Web traffic. Server logs from CNN, CNet and other sites show that the busiest times are during work hours on weekdays, with peaks at lunchtime. Data from MarketWatch's ad-serving software puts that site's total number of users at "close to double" the Media Metrix figure, says MarketWatch president and CEO Larry Kramer, who figures the sizeable difference comes from the under-representation of working surfers.

The discrepancy between internal server logs and Media Metrix became such a popular subject within the advertising world last year that FAST, the Future of Advertising Stakeholders, undertook a data reconciliation study to make sense of the divergent numbers. Bruce MacEvoy, director of advertising research at Yahoo and a FAST member, will deliver a report this week comparing panel results to log files from 30 sites -- including Yahoo, Microsoft and Time Warner -- in conjunction with the @d:tech.SanFrancisco advertising conference.

"Discrepancies vary based on a given site's attributes, but a big one is the proportion of users who come to the site from work," says MacEvoy. "No matter what they say about home and work panels, [ratings firms' statistics] are best thought of as home-user samples," he says.

At MarketWatch, Kramer says the lack of workplace measurement "is a big issue." Like everyone in the business, he wants reliable statistics that not only rank his site up high in the Internet lineup but also help measure trends over the course of the site's own history. What he gets, however, is skewed information. Considering that less than one-quarter of the overall Media Metrix panel is surfing from work, and the majority of traffic to the MarketWatch financial information site is during work hours, Kramer says, there's "a large margin of error."

Media Metrix's numbers are chiefly used by advertisers setting budgets, and that gives these discrepancies real impact. "It's data that is methodologically flawed, but people who spend money use it to spend money," says Alexander Hughes, group circulation manager for CNet Online. "It has a huge impact on revenue" for sites across the industry, he says.

Next page | We know the numbers are flawed, but use them anyway