Backscatter: 

Inside Peer Review

by Donald Christiansen

If you want to get a paper published in a journal of some repute, you will encounter the peer review process. To the neophyte, peer review might seem a jungle; to others, a familiar hurdle to surmount. In simple principle, peer review is intended to let only the worthy papers survive, matching the worthiest of them to journals of the highest reputation. Its sometimes forgotten objective is to favor the ultimate reader.

The power player in peer review is the publication editor, who selects reviewers and defines acceptance criteria. He or she usually wants the reviewer to comment on the premise of the paper and address these questions: What’s new, and how new is it? Does it follow logically from previous work? Is it believable? Are experimental results well documented? What’s missing? If it is a review paper, is it well organized and historically accurate?

Three reviewers seem to be the accepted norm, although in my own experience at IEEE Spectrum I sometimes used as many as 20 reviewers for a staff-generated, multi-viewpoint article. Unless a paper is in the editor’s own field of interest, he cannot be an arbiter of its technical suitability, but will instead rely on expert reviewers. Invited reviewers who are insufficiently qualified are expected to disqualify themselves.

Should an editor find a submitted manuscript far off the mark, he can, if the publication's policy permits it, reject it without review. He might be tempted to follow the lead of Harold Ross, the brash founding editor of The New Yorker, and simply dismiss the manuscript with the line, "I regret to say we did not like this piece well enough to use it.” But the journal editor is likely to feel obligated to let the author of an inferior manuscript down more easily, even if with an invented apology. An editor of a trade publication known to most of us would say something like this: “We have received so many articles covering essentially this same topic that we cannot accept another, excellent though it is.”

On the other hand, if an editor finds it necessary — by policy perhaps — to have a manuscript that he finds unworthy of publication reviewed nonetheless, he may resort to reviewers likely to agree with him. The process, of course, will be a sham; the editor may even alert the reviewers that he expects no more than a superficial seconding of opinion.

In general, such exceptional tactics are unnecessary. Rather, both editor and author hope for thoughtful, unbiased reviews that will help the author correct errors and extend the paper’s arguments, and so enhance its worthiness for publication.

Blind Reviews

A journal may have a policy of blind reviews — that is, the author will not be told who the reviewers are. A double-blind approach keeps the author’s identity from the reviewers as well. In narrow specialties, however, authors and reviewers can sometimes easily guess one another’s identities.

An editor may send reviews to the author with or without comment, or, as I sometimes did when editor of Spectrum, send the author only excerpts he deems pertinent. Also, a reviewer might sometimes volunteer to speak directly with an author — an offer I would invariably accept. In the same vein, when I received a comprehensive, detailed review, I might encourage the reviewer to identify himself to the author so they could communicate directly.

The Job Doesn't Pay Well

Reviewers don’t have an easy task and usually are not paid for their efforts. Their recompense may come only in what they learn that might enhance their own work. Editors cannot expect reviewers to duplicate research that involves experimentation; at best, they can expect comments on the research methodology. At the same time, reviewers can’t be expected to uncover falsifications in reported results. They can, however, confide their suspicions to the editor.

Bias and Rejection

Editors and authors sometimes suspect a reviewer of bias, to put it kindly. A particularly vicious review might suggest to an editor that a reviewer has fathomed who the author is. (Perhaps the author once voted against tenure for the reviewer!) Most editors will discount a vituperative review and seek a replacement.

Academics admit that reviewers often denigrate papers that are based on innovative work but are counter to accepted wisdom; such reviewers seem to want to avoid the publication of contrary scholarship in “their” journals. As one of them put it, those who review essays for inclusion in scholarly journals know their function is to take exciting, innovative and challenging work by younger scholars and find reasons to reject it. I think this is overly cynical, especially in our fields.

Nevertheless, skeptics say with some justification that an author has an advantage if he has been published in the journal before; has co-authored a paper with one of the reviewers; or has favorably reviewed a paper of the reviewer. Furthermore, an overburdened editor feels safer dealing with an established author.

Lest I too firmly implant the notion of peer review as fraught with pitfalls and gamesmanship, I should say that I believe most editors, reviewers and authors respect the process for its value in assessing the quality and accuracy of professional journal content.

Electronic Publishing Reviews Are a Different Matter

Too many variants exist in the time-honored process for print journal peer review to cover in this brief discussion. And many innovative review procedures to deal with the proliferation of electronic publishing are under study. William Arms, a computer science professor and part of a Cornell University team developing the National Science Foundation’s new digital library for education in the sciences, decries the “junk” on the web. He notes that while major libraries harbor materials so inferior that it is hard to believe they were ever printed, the problem is even more severe on the web, where “the barriers are very low; anybody can be an author and anybody can be a publisher.”

We hope to discuss some of the approaches to peer review of web-based publishing for a future column.

Resources

For more about peer review, see:

Harnad, Stevan, “Rational Disagreement in Peer Review,” Science, Technology and Human Values, Vol. 10, p. 55, 1985.

Arms, William Y., “What Are the Alternatives to Peer Review? Quality Control in Scholarly Publications on the Web,” Journal of Electronic Publishing, Vol. 8, No. 1, August 2002.

---

Donald Christiansen is the former editor and publisher of IEEE Spectrum and an independent publishing consultant. He can be reached at donchristiansen@ieee.org.