The posting below, reprinted with permission from the November/December 1999 issue of Academc, the magazine of the American Association of University Professors, gives excellent advice on conducting peer reviews. The author is Stephen Chilton, associate professor of political science at the University of Minnesota-Duluth. He would like to thank Bob Evans, Jim Fetzer, Mark Gonzalez, Mike Peters, Fred Schroeder, and Judy Trolanderfor their support and intelligent commentary on this article.
UP NEXT: Casting Light on Old Notions: A Changing Understanding of
Community College Faculty
------------ 1,695 words -----------
THE GOOD REVIEWER
by Stephen Chilton
Peer reviewers get little formal guidance. Following a few simple rules can make the process more useful to authors and editors alike.
Peer reviewers have a lot of power. They help editors of academic publications decide what gets published and thus indirectly help deans decide who gets tenured. But editors and deans are not their only audience; peer reviewers also influence authors. Reviewers can offer feedback to authors, enabling them to improve their work, and they can also give authors recognition: a sense that the enterprise in which they are engaged has merit, regardless of the virtues of their article. Such recognition is critical for junior authors, but even senior authors value it. And our disciplines can only benefit from an atmosphere of mutual appreciation of one another's efforts.
Yet despite its importance, peer review tends to be uneven in quality. Probably most authors have had the frustration of getting mere summary comments from a reviewer who did not seriously address a manuscript's argument-or even understand it. These problems arise because peer review is hard to do well. Here are a few of the difficulties that can disrupt the process:
* ACADEMIC COMPETITIVENESS. Most peer review is done by people with some stake in the outcome. Keeping the author's identity from a reviewer cannot disguise the intellectual impact a work will have on an academic field. The reviewer may recommend against publication of a manuscript that he or she finds threatening.
* LACK OF TRAINING AND RECOGNITION. I received no training to review my peers, and I doubt many people do. Graduate school teaches us to find and criticize logical flaws, but peer review requires far more than that, as the guidelines below should show. Moreover, faculty members receive little professional recognition for peer-reviewing, regardless of how well they do it.
* DIFFICULTY OFDEALING WITH ORIGINAL WORK. The more original the work, the more difficult it is to evaluate. It may use new methods, test the dominant paradigm, or even employ standards of a new paradigm. Those who review such works always face the problem of how to judge the new by the standards of the old.
In the hope that we can all thread our way through these problems, I offer below some guidelines and suggestions I've found useful in my own reviewing. Certainly, I would like my own work to be reviewed along these lines.
BASIC STRUCTURE OR THE TASK
The peer reviewer is responsible to both the editor and the author simultaneously. Some reviewers feel they must defend authors against censorious editors or editors against inferior manuscripts. But the task of reviewing, properly conceived, involves no such conflict. Both the editor and the author benefit from a straightforward appraisal of the manuscript.
It is useful to start by reading a manuscript's introduction and conclusion, as well as the abstract (if it has one), since your task includes judging the work's logical flow. But do read the entire manuscript. That should be obvious, but at least one journal editor has turned in desperation to excising all abstracts, because too many reviewers read only the abstracts.
Draft manuscripts are more difficult to read than published works, so peer reviewers should expect to work harder than reviewers of published works. The author is, however, responsible for providing a readable manuscript. If the author has not done so, I feel little obligation to finish reading it. I write a short, Polite, but definite statement about the writing problems, suggest that the author seek the help of a freelance copy editor, and pass along whatever useful comments I can on the work's argument.
In addition, I would suggest that you confine your comments to the manuscript; don't review the author, and don't be afraid to say no.
Avoid second-guessing what the journal needs or the editor wants. The editor gets to decide what to publish; a good editor wants your professional judgment.
Take sufficient time to review the work thoroughly, but return your comments no later than three weeks from receipt of the manuscript. If you cannot commit to doing so when you receive it, either return the manuscript immediately or ask for an extended deadline. GOLDEN RULE
Admit (if only to yourself) that your review gives your judgments, not god's. Though your editor needs you to explain your judgments clearly, you can still phrase them in a way that acknowledges their fallibility. The golden rule is appropriate here, not in a pollyannaish sense but rather in the sense of asking yourself, "given that this manuscript has problems x, y, and z, how would i like to be informed of these problems if the work were mine?"
The anonymity of reviews seems to encourage abusive behavior. I value anonymity as an aid to detachment. Still, I start each review by typing in my name, even though I remove it from the final version. I know logically that the author won't see my name, but its presence on my drafts keeps me psychologically attuned to being respectful.
State appreciatively what is good about the manuscript. Every work has something good about it: new data, a new approach, interesting insights, a breadth of perspective, grace of expression, and so on. In particular, recognize and state appreciatively the importance of the problem the author is trying to address. Even if its significance is unclear to you (and possibly even to the author), authors always strive to speak to something important. Encourage that.
State the author's main point, even if you don't find it earthshaking, well supported, or correct. Doing so assures the author and the editor that you have understood the point (or if not, where exactly you went astray), and it assures you that you've come to grips with the manuscript.
Don't focus on minor problems, even if you mention them at the end of your comments. 1, for example, get waspish over such minor problems as the use of poor punctuation or the confusion between "which" and "that." I must consciously keep myself in hand so as not to let my irritation obscure the real problems of the paper.
Insofar as possible, comments should include suggestions on how to overcome any problems in the manuscript. Sometimes problems are simply unavoidable; the choice is between problematic research or no research at all. (in my field, comparative politics, researchers using cross-national data almost always face this choice. I expect that other fields face similar difficulties.) If the problems are so severe that the research results are more likely to be misleading than helpful, then say so, and explain why. But if the results seem likely to be helpful in spite of flaws, ask that the problems be acknowledged, their likely biases assessed, and means for overcoming them in the future considered.
One common type of nonconstructive comment is namecalling or labeling: characterizing without describing. Here are two real examples (neither about nor by me) of reviewers' name-calling: (1) "Gives the impression of being written by an amateur without knowledge or training in the field." (2) "This looks to me like a high-quality undergraduate term paper, or something written for a popular magazine. It is a clearly written surface-level summary of the main threads of [research area x] and [research area y] over the years." Such comments are hurtful without being useful. Avoid them.
Distinguish your objections from Your queries and suggestions. Objections say, "I believe this to be a problem that must be dealt with for the manuscript to hold together." Queries point out minor problems, such as confused meaning or omission of topics the reviewer merely wishes the author addressed. Suggestions are simply alternatives for the author to consider. The author needs to know what's essential to address.
If the manuscript is inappropriate for the publication to which it was submitted, don't abuse the author; just suggest other, more appropriate forums.
Since thomas kuhn published The Structure of Scientific Revolutions in 1962, it has been well known that all disciplines, even the physical sciences, are structured and divided by competing paradigms that cause serious problems of mutual incomprehension for their respective advocates. This phenomenon frequently puts the reviewer in a quandary. A reviewer might legitimately hold that an author's entire line of research is meaningless, since the author fails to address what the reviewer sees as the critical issues in the field, and reviewers cannot take the relativist's escape that everyone has his or her own approach. Many reviewers act as if they have privileged access to the truth. As psychologist e. Rae Harcum pointed out in Imperial Review, such reviewers act on the principle that the evaluator knows more than the author. But this stance is not philosophically supportable and, moreover, it is harmful-both for the reviewer emotionally and for a discipline.
Reviewers who remain silent about their position when they disagree with an author's line of research either acquiesce in what they believe to be the author's error or disguise the' real roots of any criticism they offer. When I differ profoundly with an author's basic orientation, I allow myself to take one paragraph to explain why I believe my perspective is better than the author's, after which I attempt to review the manuscript on its own terms (but pointing out specific places where the advantages of my perspective are particularly apparent). Thus both the editor and the author can evaluate the review and the manuscript in light of this acknowledged difference.
I take the same approach in matters of taste. For example, I dislike the stilted, rigid structure of "introduction, hypothesis, methods, data, analysis, conclusion." When reviewing manuscripts structured in this way, I state my position in a concise paragraph at the beginning of my review, where the editor will be sure to see it, and then let it pass.
Good peer reviewing isn't difficult if one starts with the proper attitude toward the author, namely, that we're in this together. Peer reviewing isn't a contest to prove who's smarter, still less a fight against an enemy. One of the nice things about not being God is that one isn't personally responsible for the whole universe. If we do our best, without fear or favor, then probably our disciplines will do well, and we can leave the final outcome to some higher power.
ACADEME November-December 1999 55