Skip to content Skip to navigation

Human Reviewers: The Achilles Heel of Scientific Journals

Tomorrow's Research

Message Number: 
258

Folks:

The posting below looks at ways to improve the journal review

processing in light of increasing time demands and numbers of article

submissions. It is from a keynote address at the 5th Internet World

Congress for Biomedical Sciences

December 7-16, 1998, by Floyd Bloom, (Department of

Neuropharmacology, The Scripps Research Institute. Reprinted with

permission of the author. Further information can be found at:

http://www.mcmaster.ca/inabis98/keynote/bloom/two.html

Regards,

Rick Reis

reis@stanford.edu

UP NEXT: Some Grad Students Make Terrific Teachers

Tomorrow's Research

 

----------------------- 1,533 words ----------------------

HUMAN REVIEWERS: THE ACHILLES HEEL OF SCIENTIFIC JOURNALS

Contact Person: Floyd Bloom (fbloom@SCRIPPS.EDU)

Bloom, Floyd (Department of Neuropharmacology,

The Scripps Research Institute, USA)

 

Abstract

The World Wide Web and other forms of digital communication have lead some

to predict the demise of printed journals. In my experience, scientific

articles can be more efficiently reviewed and edited by digital document

sharing. High quality print journals will remain the preferred scholarly

venue for authors's best works, but only if the performance of human

reviewers can be improved and appropriately rewarded.

-------------------------------

The Review of Scientific Manuscripts

Participants in this Conference are among the most experienced of

electronic browsers for scientific content. Yet unquestionably, almost all

of them are still enmeshed in a process of scientific manuscript submittal

to their favorite journals that has been the standard for more than 200

years. As shown in Figure 1, an author submits a manuscript to a journal by

delivering it to that journal's editor. The editor, having decided through

various means - often introspectively and subjectively - that the paper

would be suitable for that journal's mission, then ponders several

questions. Who is a peer of this author, in content area, in technique and

when possible in experience? Which reviewers should be avoided for

competitive conflicts and past disputes? How much of the submittal's data

can the editor expect the reviewer to examine in detail? For example,

should the statistical conclusions be re-calculated? Should all the raw

data from which the observations to be reported were extracted be made

available to the reviewers? In short, what does the editor need to know to

decide whether to accept or reject a submittal. The following personal

observations are based on my experiences as the editor or reviewer for

several scientific journals, including service for the past 4 years as

Editor-in-Chief at SCIENCE.

--------------------------------

Figure 1: The standard procedure for manuscript submittal and review. An

author selects a journal (1) by transmitting a submittal to that journal's

editor. The Editor selects reviewers (2) who return their criticisms and

suggestions to the editor, who then reinforces their views in

correspondence to the author (3). Eventually, after one or more such rounds

of review and revision, the editor will reject or accept the paper. The

accepted paper is then transmitted to the publisher (4), who then includes

the paper in the journal (5) as the production process allows, pending copy

editing, return of author galley proof corrections, and barring

complications in graphic reproduction.

-------------------------------

The Review of Scientific Manuscripts Can Be Wholly Electronic

Until quite recently, each of the steps of the scientific manuscript review

process--submittal, review, revision, acceptance and eventual transmittal

to the publisher-- required physically transporting documents, tables and

figures from one player to another (see figure 1). For the past year,

Warren Young and I have been conducting an experiment with Elsevier

Scientific Publishing in which this entire process is conducted

electronically through the World Wide Web without any hard copied

materials, and without physical manuscript required. The SMART process

(System for Manuscript Review and Authoring Technology) and the required

authors' transmittal module are available on the Web at the URL

http://smart.scripps.edu:8000/smart.html. The editor's choice of peer

reviewers is aided by searching a database of qualified and self-declared

reviewers on the basis of keywords in their areas of content and technical

expertise Accepted papers are eventually transmitted to the publisher,

posted on their journal Web site within a few days, and eventually printed

in the appropriate section of the journal Brain Research. Other systems for

electronic submittal and review are already in existence in Astrophysics.

The journal Pediatrics, and recently, the Journal of Neuroscience have

committed to digital only 'publication' of papers submitted conventionally

to their review process. Clearly, the review and the dissemination channels

of scientific publishing can profit from digital document sharing.

Why Do Authors Select Certain Journals for Their Best Papers?

The question of technological efficiency in the review process is but one

element of the multi-faceted selection process by which authors decide the

journal on which they wish to bestow their latest manuscript. As shown in

Figure 2, authors select their journals on the basis of the past content

of the journal in regard to a variety of factors in peer appreciation,

timeliness of the manuscript selection and decision process, and the

overall quality of the appearance , publication, and delivery of the

journal. The reputation of the journal for its previous quality of papers

and content is critical. How often does it appear? Who else has published

there? How many subscribers does the journal have? Is it listed in major

indices of publication? How many libraries carry this journal? Other

aspects of peer appreciation Include the speed with which its submitted

papers are accepted for eventual publication, and the quality of appearance

of the final manuscripts and artwork. In short, authors select journals

for their works for the impact these publications will have on their

scientific reputation for academic promotion and for support of their grant

applications.

-------------------------------------------

Figure 2: Factors in journal selection by authors. Authors select journals

in part by bearing in mind several factors as noted in the text.

-------------------------------------------

Reviewers Are the Achilles Heel of the Review Process.

Almost all aspects of the scientific manuscript and review process can be

accelerated by digital transmittal of manuscripts, illustrations, reviews,

and revisions between authors, editors and reviewers. However, in order to

maintain the apparent quality of their journal, editors seek the most wise

reviewers on the basis of their past performance and for their scientific

reputations as scientists. Often reviewers in 'hot' areas of science may be

in great demand for reviews by many different journals. Reviewers working

in relatively quiet research fields may also be in equally intense demand

because the pool of qualified reviewers is quite small. Regardless of the

reasons, only the most dedicated academic scholars undertake to review.

There are many possible reasons for their participation in the process, but

>none have been studied scientifically.

>From my personal perspective, some factors are suggestive: past pleasure in

the scientific publication process, respect for the scientific review

system, and anticipation of equally rigorous and fair reviews in the

future. Often the 'best' reviewers are under their own pressures, such as

research, grants and their own papers. No matter how quickly manuscripts

and reviews may be digitally exchanged between editors and reviewers, the

major temporal sink remains the delays in return of useful reviews from

those asked to review. Journals with lesser reputations may suffer the

most, since the needed reviewers will owe little allegiance to the journal,

and the quality of their reviews may reflect their judgement of that

journal's reputation. Delays in receipt of reviews with useful critical

judgement may in turn delay the decision making, which in turn affects the

journal's aggregate reputation. All in all, prompt, high quality reviewing

is the essential element of the scientific manuscript review process, and

the one most vulnerable to the vagaries of journal editor-reviewer

interactions.

----------------------------------------

Figure 3: The critical role of reviewers. Editors select reviewers not only

for their scientific expertise in certain areas of content and

technology, but also for their past performance as reviewers in terms

of open-mindedness to new ideas, including data which may overturn

the reviewers' own findings. Editors also select reviewers for their

capacity to respond with reviews to papers in a

timely manner. While the authors, the publishers, and the editors are all

compensated for their performances in this process, the reviewers in

general serve only in anonymity, with the gratitude of all

----------------------------------------

Conclusion: To Improve Quality and Speed of Review Requires Rewards for

Reviewers

Given the importance of reviewers in the scientific manuscript review

process, how can editors and their publishers offer incentives for

exemplary performance. Monetarily, it is possible to provide free

subscriptions to reviewers. That reward will have the most value when an

expensive journal is worthy of the subscription charge (e.g., in terms of

content quality, amount and timeliness of production). However, free

subscriptions may be constrained by the goals of the publisher, whether

they are commercial or not-for-profit. Dedicated scientists reviewing for

their society -owned journals may already be subscribers as part of their

membership fee. Generally, the monetary value of a free subscription may

well not reflect the time and effort required to review frequently. Editors

can offer reviewers prestige, but appointing them to visible editorial

positions, or inclusion in annual lists of reviewers, but this notification

my feel to reach the members of appointment and promotions committees. One

incentive rarely practiced is for editors to write to departmental chairmen

to praise their faculty member's performance as a reviewer. Editors can

offer reviewers some insight into their decision making process by

providing them copies of the letters of transmittal from editor to author

after the editor has read the reviewers comments.

By showing reviewers which kinds of assessment are most valuable in shaping

a paper in need of revision, an editor can achieve several important goals.

The editor can

help educate reviewers to the quality standards of the journal and to the

sorts of information that editors need to make their decisions to accept or

reject. Lastly, it should be possible in the near term to offer potential

reviewers, up-to-date and comprehensive lists of pertinent bibliographic

materials related to the manuscript under review. Whether any of these

possible rewards will be appreciated is also an experiment in progress.

Nevertheless, the reviewer has been the most critical, and yet absolutely

under-valued element of the scientific manuscript review process. Rewarding

good reviewers for excellent performance is essential for maintaining the

quality of the review process, no matter how quickly manuscripts are moved

from authors to editors to reviewers.