Skip to content Skip to navigation

Contending with Reviewer’s Comments (about research proposals)

Tomorrow's Research

Message Number: 
1450

This chapter is about investigators’ behavioral and emotional responses to not being funded and suggests strategies that might help keep their research program on course.

Folks:

The posting below looks at a variety of helpful response approaches to feedback from reviewers of research proposals .  It is from Chapter 2, Contending with Reviewer Comments, in the book The Research Funding Guidebook: Getting It, Managing It, & Renewing It, by Joanne B. Ries and Carol G. Leukefeld.  SAGE Publications, Inc. 2455 Teller Road Thousand Oaks, California 91320 E-mail: order@sagepub.com. Copyright © 1998 by Sage Publications, Inc. All rights reserved.  Reprinted with permission.

Regards,

 

Rick Reis

reis@stanford.edu

UP NEXT: Forms of Intelligence

 

Tomorrow's Research

---------- 2,344 words ----------

Contending with Reviewer’s Comments

(about research proposals)

 

Investigators’ reactions to comments about their application may be unique in the ebb, flow, and intensity, but their substance is similar. When an application is funded, the mood is one of elation moderated by reaction to the negative comments, which inevitably are included. When an application is not funded, the mood is disappointment, frustration, and sometimes discouragement and resignation. This chapter is about investigators’ behavioral and emotional responses to not being funded and suggests strategies that might help keep their research program on course.

Reestablish Control

It is our recommendation that before responding to the reviewers’ comments an investigator recall the events that led to the application’s submission. This reminiscing can clarify the role the application was perceived to play in their career plans and can rekindle motivation. These memories can provide a platform from which to evaluate their past decisions and inform the decision about continuing the application process.

Reviewers’ comments usually hold surprises for investigators. Sometimes when investigators learn that they have not been funded, they quickly conclude that it was a specific factor that reviewers did not like.  When an investigator can identify a weakness, a sense of control can be maintained, and the investigator’s scoping systems remain in good order. The review comments, however, might be very different from those anticipated, and the investigator is left to deal with criticism for which he or she is unprepared.

An expected characteristic of review comments is the unrelenting demand for scientific rigor, theoretical relevance, and important societal implications. Whether the competition is in-house, regional, or national, reviewers generally demand a high level of scientific rigor.  Sometimes investigators believe that the demand of some reviewers for detail in the project design, analysis section, and comprehensiveness in the literature review are excessive – even bordering on being picky. Minor criticisms can evoke a variety of emotional responses, and suggestions of inadequate scientific rigor in the project plan or that the project has little theoretical importance strike at the heart of an investigator’s self-image as a scientist.

Professional careers seem to be fraught with opportunities to receive criticism. Consequently, it is important that investigators develop coping mechanisms for dealing with criticism, and often strategies for doing so are actively sought. The effectiveness of these strategies frequently depends on the source of the criticism, the personal relevance of what is criticized, and whether the criticism appears in a public forum.  With experience as the target for criticism comes the ability to ignore criticism that is unfair or unjustified and to search diligently for the value in criticism from an informed source. Reviewers’ comments are particularly difficult for investigators to deal with because they have a strong belief in the importance of their project and they might perceive reviewers to be important sources of information. This already difficult situation is made more difficult because the usual coping mechanisms developed for dealing with criticism in professional life are often different from what is required if an investigator wishes to revise and resubmit an application. Reviewer comments, therefore, have the potential to push investigators beyond their usual coping systems. However, if investigators wish to resubmit an application, they cannot retreat from or ignore the comments. The only choice they have is to respond to each of the criticisms whether it is perceived as being unfair, unjustified, or arising out of misinformation.

Develop an Adaptive Behavioral Response

Receiving news that a project has not been funded can influence multiple aspects of investigators’ professional lives whether they are clinicians, basic scientists, community practitioners/scientists, or administrators. In some cases, it may not matter how close the project’s score was to the funding payline because project complications can arise from delays incurred with a resubmission. For example, those who had arranged to free up time for the project may now have to reschedule that time as well as free up future time if the project is to be resubmitted. For investigators who were relying on the funding to keep their program going, the delay of another funding cycle might place their program in jeopardy. For others, being close to the funding payline can be significant. If, for example, the not-funded application was a second or third resubmission and was given a poor score, an investigator may have to face a decision about continuing the particular line of investigation or seeking an alternative funding source. Whereas if the score is excellent or outstanding, the time and effort of resubmitting could be well spent.

A project application eventually involves a spectrum of people (e.g., administrators, collaborators at other sites, colleagues, administrative staff such as department heads/chairs, deans). Many of these people often meet with the principal investigator (PI), attend the same professional meetings, and possibly are collaborators on other projects. In view of this small but influential community that evolves from a project application, it is critical that investigators, whatever their internal state, maintain a public appearance as stable, rational individuals who learn from their experiences.

We have known investigators who, on receiving news that their application was not funded, essentially stopped their research careers. Sometimes this was their intention. They were not able to maintain their coping mechanisms or could not find answers to questions posed by the reviewers and eventually decided that having externally funded projects was not desirable. Others may, by their behavior, give the impression that they have resigned. Their colleagues perceive them as not being intellectually responsive, as using their time for other pursuits, and as no longer being reliable consultants and collaborators. To prevent such an impression, it is imperative that investigators continue to interact with their colleagues and, if anything, make their ongoing projects more visible, however serious the disruption is from not being funded. It is important to remember that many careers require external funding to sustain advancement and that most applications require collaborators. Until investigators definitely decide to segue out of the external funding stream, it is imperative that they continue to behave as competent and reliable investigators who are able to complete projects and overcome barriers. It is only in this way that investigators can successfully negotiate with colleagues for their valuable time and in turn continue to be considered a valuable asset on their projects.

Develop an Adaptive Psychological Response

Investigators have different strategies for developing an adaptive psychological stance to deal with reviewer comments. One common strategy is to read the comments, get the gist of them, and then put them out of sight for a few weeks. This is not to be viewed as an act of procrastination. Investigators faced with reviewer comments are on the threshold of making important decisions. Putting the comments aside can give them time to come to terms with being criticized and to get better acquainted not only with the nature of the criticisms but also with the reviewers’ perceptions of the application’s strengths.  This strategy helps to ensure that the decisions made about the project are based on its value, not on the intensity of fluctuating emotions. 

Fuller wrote about the course through which an investigator’s emotions might flow after reading the reviewers’ comments for the first time. She called this course the “pink sheet syndrome” (Fuller, 1982) after the color of the paper on which the reviewer comments were then printed.  It seems that emotional detachment is difficult to maintain when an investigator’s favorite ideas and professional abilities are being criticized. Even when a proposal receives a fundable score, it is difficult to read the reviewer’s negative comments with equanimity.  When, however, the score places the proposal in the not-funded, emotional detachment is not only difficult but usually impossible to achieve.  This reach is understandable considering the amount of intellectual and administrative work embodied in a completed application. Usually, investigators have battled with competing theories, designs, analyses, and data sources before arriving at the product that was reviewed. Investigators become intellectually attached to the final project, are convinced of its significance, and are hopeful that they have communicated their vision and their enthusiasm. In addition to investing creativity, the investigator has endured the mundane labor of producing the application forms and presenting the project plan according to funding source instructions – an activity rarely done under conditions of calm confidence. For some investigators, the mere thought of repeating this activity is onerous. In spite of this, the review comments must be read and evaluated, and ultimately responded to with emotional neutrality.

The negative comments make the biggest impression the first time the reviewer’s comments are read. Fuller (1982) calls this Stage I: the onset of anxiety and panic. During this stage, the investigator places exaggerated importance on comments that seem directed at the viability of the project. Sometimes serious thought is given to abandoning the line of investigation and “moving on to something less demanding.” The counterpoints that investigators find during this stage are the positive comments often provided. Although these comments seldom are sufficient to quickly restore an investigator’s emotional equilibrium, they do provide encouragement and inhibit precipitous abandonment of the project.

Stage II, according to Fuller, can take the form of depression, shame, or anger. Investigators are besieged by doubts of their ability to think and to achieve success, shame about what others will say when they learn that the project has not been funded, and anger at the funding source for failing to understand the value of the proposed project.  During this period, because it is fraught with emotional uncertainty, we suggest that investigators resist making public statements about reviewers (investigators do not know who knows whom), calling members of the review panel (usually against the rules), or calling program officers (unless a prepared script is used and emotional equilibrium restored). Cultivating a demeanor that makes a good impression on colleagues and the funding source representatives is as important during this phase as it was when the initial contacts were made.

The key to career longevity for many investigators is to develop a psychological stance that enables them to revise the application and to benefit from the experience. A good way to maintain a positive attitude is to remember how the project was thought about during its early stages of development. Recognize that the predominant theme during that time, whether it was stated or not, was risk taking. Science requires intellectual risks such as public discussions of new ideas and submitting new ideas for competition. Investigators recognize that not taking risks can lead to intellectual stagnation. Expanding a knowledge base requires investigators to take risks knowing that inherent in this pursuit is the potential for failure, and occasional failure should not be surprising. Failures, when put to work, can be the foundation for future successes. Investigators, to take advantage of these opportunities for future success, need to be teachable and be open to all learning opportunities.

A unique educational opportunity is offered in the review comments. They could be considered to be courses designed by national experts as a guide to future work. If investigators ask themselves, “Would I have sent this application to an eminent scientist in my field for review?” most would answer no. (If the answer is yes, it is hoped that it was sent before submission and that comments were incorporated in the application.) In fact, that is exactly what investigators do when they submit an application for funding. From that perspective, the most rational action for investigators is to make the best possible use of the opportunity for career development by using the reviewer comments in the revised application.

The emotional response to the second reading of the comments is usually less intense and is often quickly followed by asking colleagues for opinions about the comments. To reap the greatest advantage from reviewers’ comments, investigators need to get help from someone who has already been through the process, responded to comments, and is alert for information gleaned from “reading between the lines.” Sometimes reviewer comments have been compiled by staff from the original reviewer statements and may have been edited to keep the tone from being harsh. Other times an investigator receives the comments exactly as written by the reviewer. The second method provides a better opportunity to understand how each reviewer saw the project, and it enables following their line of thought. In many instances, it also provides the investigator with better insight into where the communication between the reviewers and the investigators went astray.

Sometimes the overall project will be discussed in positive terms and perhaps specifically, for example, its value or the value of the unique data to be collected. These comments are to be interpreted within the context of the other positive and negative comments that follow. The introductory comments should be evaluated in terms of what they say and do not say. Although the absence of comments about the value of the project, the data collected, or the scientific endeavor need not spawn discouragement, it does need to be taken seriously. Investigators need to determine if there has been an oversight or if the reviewers are actually discouraging further project development. One way to get a better understanding of how to use these opening comments is to read comments received by a colleague. The objectivity inherent in this situation allows an examination of the pattern of reviewer comments, an understanding of the way negative and positive comments are developed, and how these might have been weighted during the decision-making process.

The weight given negative and positive comments is usually reflected in the application’s score. It is this score that needs improving. This is best done by realistically assessing the positive comments and making plans to respond to the negative comments. Sometimes, however, there are very few or very minor criticisms with not many positive comments and a poor score. This kind of review usually represents a situation in which, for some reason, the reviewers react negatively to the application and were unable to state their objections.  In this situation, an investigator needs to contact the program official by telephone to try to determine how the disparity between the score and the comments arose.

 

References

Fuller, E.O. (1982). The pink sheet syndrome. Nursing Research, 31(3), 185-186.