Gut Check Gone Wrong? GamePolitics Discounts Number of Bisexual Women Found in EQII Study

GamePolitics was more than just a little skeptical about the results:

From here, the bisexuality figure alone renders Caplan's study questionable. EQ II women are five times as likely to be bisexual? What is the statistical likelihood of that occurring? Perhaps the answer lies in the fact that the survey relied on self-reporting data from EQII players. Could some respondents have been pulling Caplan's chain?

Perhaps. But before calling the entire study into question, it's important to recognize an important fact. Though much of the info the study gathered was indeed based on "self-reporting," the EQII study is the first to correlate its findings with player behavior data supplied directly by an MMO publisher, in this case, Sony Online Entertainment.

That data came primarily in the form of detailed stats for hours played on accounts whose owners participated in the study. Such records are meticulously tracked by publishers and are, in general, highly accurate.

In the EQII study (a cconference version of which is available here), both men and women consistently under-reported the amount of time they played each week. Men were off by about an hour, while women underestimated their play by about three hours. Both differences were statistically significant.

However, gameplay stats for players who claimed they were bisexual women found no statistically significant difference between their estimates of weekly playtime and actual gameserver records. That is, they were more accurate than both women in general and men. This puts the respondents who identified themselves as bisexual women in a statistically unique group and seems to challenge GamePolitics' assumption that they must have been lying.

GamePolitics also relied its own editor's anecdotal experience in challenging the study's findings. But while the editor of GP is a known World of Warcraft enthusiast, the study in question analyzed a completely different virtual world, Everquest II. Comparisons across these worlds, particularly those based on anecdotal evidence, are unlikely to be valid, given that the two games have vastly different demographics and cultures. As one poster on Terra Nova, an academic blog for the study of MMOs, noted:

WoW has reached a level of populist appeal that EQ2 never has. Blizzard advertises WoW on major network television and has a wide swath of online adv[ertising across social Web platforms...

[World of Warcraft] may have significantly different patterns on why someone picks up the box, and correspondingly different demographics as it makes its way through different social networks that EQ2 can't get into due to "nerd resistance."

It's also worth pointing out that the study in question was conducted by some of the leading researchers in the field. In addition to Scott Caplan of the University of Delaware, who was quoted by GP, the team included Dimitri Williams, Mia Consalvo and Nick Yee. Williams is a noted sociologist who served as an expert witness in the case of ESA v. Blagojevich. He's also the author of a study that found no link between violent games and aggression. His testimony countered that of "media effects" researchers like Craig Anderson. Consalvo has spent her career studying the evolution of culture and social networks in online gaming, and Nick Yee created the Daedalus Project, which to date has collected survey data from more than 40,000 virtual world residents. These people are hardly noobs.

So while gut reactions are valuable, we have to be prepared for findings about games and culture than run counter to our expectations and intuitions. Virtual worlds, in particular, are poorly understood, and it's likely that other, equally startling, conclusions will emerge from future research.

Don't get me wrong. As Williams readily admits, "the social sciences rarely prove anything." And it's entirely possible respondents were "pulling the researchers' chain" just to get their hands on the Greatstaff of the Serpent. But if we're going to cite studies that reinforce our standing opinions about gamers and the cultures they create, we have to be prepared to accept those that challenge them as well. This is all the more true in cases like the EQII study, where the researchers are not only credible, game-savvy experts but are also correlating self reporting with cold hard server data.

At least, that's what my gut tells me.