Online game created to improve scientific peer review accuracy

Posted November 13, 2011 by Jeremy Peel in News.

An online game has been created for a study which aimed to prove that cooperation between authors and reviewers would improve scientific peer review accuracy.

It doesn’t sound like an especially fun game, in all honesty. But the John Hopkins Bloomberg School of Public Health created an online game for a study which aimed to improve scientific peer review accuracy.

The study set out to prove that peer review accuracy would be improved by closer cooperation between authors of research studies and their reviewers. Peer reviews are often carried out anonymously, and can determine whether grants are awarded or academics promoted.

The study made use of online multiple-choice question game launched from the Amazon EC2 cloud. It was played by 7–10 people – members of a research laboratory – over a fixed period, and was designed to “replicate several components of editorial peer review”.

“At any point in the game a player chose between solving problems or reviewing (accepting or rejecting) solutions submitted by other players,” states the study.

“The software also played the role of journal editor and randomly assigned submitted solutions to players for review. At the end of the game, the two players with the largest number of accepted submissions received monetary rewards, reflecting the conventional publish or perish academic incentives.”

The game was played in circumstances replicating closed peer review, and in circumstances replicating open peer review. The study found that, when the game was played publically and under open review, cooperative interactions increased 13%.

I’m not sure whether it improves the study’s accuracy, but having players vote for each other before receiving a prize for the highest score sounds like an entertaining mechanic. If it works for Come Dine With Me, maybe we can replicate it in competitive multiplayer gaming?

At the beginning of the month, a study found that playing video games boosts creativity.

For more video-game related studies, stick around at BeefJack.

Tagged as Amazon, Amazon EC2, Come Dine With Me, John Hopkins Bloomberg School of Public Health, Online game created to improve peer review accuracy, Study