1
Improving Fellowship Review: Content Analysis of Feedback from the Request for
Information
The Request for Information
NIH is proposing a revised framework to reorganize the peer review of individual National Research Service Award
(NRSA) fellowship applications under regulatory criteria 42 CFR 66.106 by restructuring the review criteria and modifying
sections of the PHS Fellowship Supplemental Form specific to NRSAs. NIH believes that these changes will allow peer
review to more fairly and effectively identify the most promising applicants.
From April 17, 2023 – June 23, 2023, NIH solicited feedback on the recommendations for improving NRSA fellowship
review through restructured review criteria and modified application instructions via a public request for information
(RFI). The RFI was publicized broadly through standard channels such as the NIH Guide Notice and publication in the
Federal Register. CSR and the Office of Extramural Research (OER) co-published a blog announcing the RFI. In addition,
direct emails were sent to leadership at almost 500 educational institutions and the RFI was publicized through many
NIH social media channels. In response to this extensive outreach, 164 unique responses received: 147 from individuals,
10 from scientific societies, and 7 from academic institutions. Responses were submitted through the RFI web form, in
response to Review Matters and Open Mike posts, or via email.
This report is a content analysis of the feedback received. It is organized thematically according to comments on
proposed changes in criteria, comments on proposed changes in the application, and comments on other topics.
Highlights
The points below capture predominant themes along with less common, but important and actionable, feedback.
• Comments from individuals and scientific societies/institutional alike were predominantly favorable.
• Regarding the proposed review criteria, the predominant view was that the proposed review criteria will
reduce institutional bias, focus review on key criteria, and reduce reviewer burden.
• Comments regarding proposed changes to the application were predominantly in favor of revising the
fellowship supplemental section of the PHS, and expressed the belief that the changes will streamline the
application, reduce burden, and improve clarity and accessibility of the application process.
• Some respondents shared mixed sentiments and noted that though the changes were positive, they do not
go far enough to simplify the application.
• There was strong support for eliminating the requirement to submit grades as part of the application,
although a small subset of commentors believed that grades should still be evaluated.
• There were significant concerns about the “optional statement of special circumstances.”
• Additional comments suggested a need to clarify the intent or practical application of aspects of the review
criteria, specifically:
o It is unclear how to evaluate “determination, persistence, and creativity.”
o The distinction between criterion 2 (Science and Scientific Resources) and criterion 3 (Training Plan
and Training Resources) is unclear. The two seem to request evaluation of overlapping
considerations.
o Better guidance and instructions are needed regarding how to incorporate information from the
application materials into the evaluation appropriately.
• Comments on the importance of the sponsor’s track record of training were split. Some thought the
proposed changes would still give undue advantage to senior investigators with extensive trainee lists, while
NIH Center for Scientific Review
13 Sept 2023
2
others thought track record was downplayed too much and that it was an important consideration that
should be explicitly considered.
• Regarding proposed changes to the application, the most common request was for additional guidance
about what information should be provided in the various application sections:
o This was expressed regarding the letters of reference, the applicant statements of qualifications and
self-assessment, the sponsor’s assessment of the applicant, and others.
• Occasionally respondents thought the application remained duplicative in (unspecified) areas and was still
too burdensome.
• Many respondents commented that good training and informational resources are vital for effective
implementation of the proposed changes. Commentors requested detailed guidance on the information
that is encompassed within each statement required in the application.
Background
CSR convened an Advisory Council Working Group (CSRAC WG) in Fall 2021 and charged them with evaluating the
fellowship review process and making recommendation to make it as effective and fair as possible for all. The CSRAC WG
gathered data, public input, and NIH input. Final recommendations were endorsed in Fall 2022 and NIH Institute
Directors approved the major recommendations and, in December 2022, called for implementation.
In the data utilized, multiple sources raised concerns about reputational and career stage bias, information utilized to
judge applicants, and a burdensome application for both applicants and reviewers. The data indicated that NIH is
potentially leaving out highly promising young scientists because of a process that too heavily favors elite institutions,
senior, well-known sponsors, and an overly narrow emphasis on traditional makers of early academic success. The
CSRAC WG proposed two major recommendations with a goal to improve the chances that the most promising
applicants, no matter who they are or where they are based, will be consistently identified by peer review. The group
recommended that NIH (1) modify the fellowship review criteria and (2) revise the fellowship supplemental section of
the PHS.
Methods
Comments from all sources were combined into one dataset for analysis after which responses submitted from scientific
societies and academic institutions were identified and separated out into a second dataset. The same coding approach
was used to tag comments for both datasets. Analysts utilized a two-step approach to coding comments. The first step
consisted of analysts reading each comment in its entirety and coding it at a relatively high-level (Favorable,
Unfavorable, Mixed Sentiment, Neutral, NRSA-related, or Not Applicable). Comments tagged as “Not Applicable”
discussed topics outside of review criteria and application instructions/materials and were excluded from the analysis.
During the second step of coding, the analysts utilized the coding scheme to tag subtopics in each comment derived
from components of the proposed framework (review criteria and revisions to the fellowship supplemental section of
the PHS). Analysts worked collaboratively during the coding process and modified the coding scheme iteratively to
account for new themes that emerged and to achieve satisfactory inter-rater reliability.
Comments on proposed changes in criteria
The first major recommendation by the CSRAC WG was to modify the fellowship review criteria – three main ideas drove
recommendation one: (1) the group wanted to better focus reviewer attention on the 3 key assessments: potential of
the applicant, strength of the science and the quality of the training plan; (2) define criteria to give less advantaged
applicants a better chance without disadvantaging others; (3) reduce bias in review by reducing inappropriate
consideration of sponsor and institutional reputation.
NIH Center for Scientific Review
13 Sept 2023
3
Overall, many individual respondents as well as most scientific societies/institutional respondents were supportive of
efforts to simplify the review criteria for NRSA fellowships to emphasize the potential of applicant, strength of science,
and the quality of the training plan. Respondents, whether individual or institutional, commented that this revised
framework is generally an improvement over the current criteria and will reduce institutional bias, focus review on key
criteria, and reduce reviewer burden. Some individual respondents shared mixed sentiments and noted that though the
changes were positive the revision did not go far enough to mitigate sponsor and institutional biases. A more detailed
summary of the comments from individuals and scientific societies/institutions on modifications to the review criteria is
presented below.
Comments from Individuals
Many respondents were supportive of the efforts to simplify the review criteria for NRSA fellowship applications from
five to three scorable criteria. They thought that this revised framework is generally an improvement over the current
criteria and will reduce institutional bias, focus review on key criteria, and reduce reviewer burden. Those in support of
the new framework made the following points:
• Most respondents were enthusiastic about the elimination of undergraduate grades from the application and
perceived grades as not having bearing on the potential of the applicant. They viewed the change as a step to
increasing equity for candidates who might have an unconventional educational background.
• Many respondents believed that the proposed changes would make reviews more equitable and less subjective
for applicants from disadvantaged backgrounds, without penalizing other applicants.
A few responses were unfavorable. Criticisms of the proposed plan include:
• The proposed plan will not substantially alter the peer review process for these applications and the revised
criteria do not mitigate sponsor and institutional bias.
• The changes do not go far enough to address perceived inequities within the NIH fellowship peer review
process.
Criterion 1: Scientific potential, fellowship goals, and preparedness of the applicant
Many respondents were in favor of Criterion 1: Scientific potential, fellowship goals, and preparedness of the applicant
and welcomed the delineation of the review of the applicant from the training plan. Respondents appreciated that the
revised criteria focus on the trajectory of the applicant’s career in light of their opportunities.
Criterion 2: Science and Scientific Resources
Many comments related to this criterion were favorable, sharing viewpoints such as:
• The elimination of the peer review of financial support will increase equity for applicants from
smaller/newer/less well-funded institutions and increase the likelihood of obtaining funding based on their own
merits. These respondents expressed the opinion that training potential and scientific merit of the application
are not negatively impacted when the mentor does not hold an R01-equivalent.
Criterion 3: Training Plan and Training Resources
There were many positive responses regarding this criterion. The following viewpoints and suggestions were submitted:
• There was support for dropping the consideration of institution, a change that was perceived as helping to
diminish institutional bias. Respondents were also supportive of differentiating the evaluation of the applicant
from that of the training plan. These changes were perceived as maintaining focus on the quality of the
applicant and on the proposed training plan.
NIH Center for Scientific Review
13 Sept 2023
4
Areas Requiring Clarification and Reviewer Training
Respondents expressed concerns about areas of the review criteria that were perceived as confusing or challenging to
review and highlighted areas that might require enhanced reviewer training before implementation. Feedback includes:
• Respondents noted that some components of this criteria (creativity) are challenging to review and more
guidance and training around the review of this criteria would be beneficial.
• Under Criterion 1: Scientific potential, fellowship goals, and preparedness of the applicant, some respondents
said that evaluation of “determination, persistence, and creativity” is extremely difficult to subjectively review
given that mentors and sponsors are not specifically asked to address this.
• The difference between Criterion 2: Science and Scientific Resources and Criterion 3: Training Plan and Training
Resources was not evident and the two seemed to be asking for the same evaluation with slightly different
wording. More definition around these criteria and their evaluation is needed.
• Some respondents viewed Criterion 3: Training Plan and Training Resources as a problematic criterion in as
there are many approaches to training and training philosophies and the evaluation of this criterion may result
in personal perspectives outweighing objective evidence. Enhanced guidance is needed for how to effectively
review this criterion.
Comments from Scientific Societies and Academic Institutions
A majority of societies/institutions were in favor of revising the five scorable review criteria into three areas: potential of
the applicant, strength of the science, and the quality of the training plan. Most agreed that the revised review criteria
would focus on the applicant’s potential growth and the merit of the sciences. They were highly enthusiastic about
removing “Sponsors, Collaborators, and Consultants” and “Institutional Environment and Commitment to Training” from
the current review criteria, believing this would reduce sponsor and institutional bias, and that the revised criteria would
provide more equity and access to the applicants from disadvantaged backgrounds or those with junior sponsors.
Criterion 1: Scientific potential, fellowship goals, and preparedness of the applicant
Most of the societies/institutions responded favorably regarding criterion 1 and supported the emphasis on the
applicant’s accomplishments and training potential. Some societies/institutions wrote in favor of the elimination of
grades from the review.
A few societies/institutions shared concerns about the proposed plan. The criticisms shared include:
• The success of the proposed changes depends on applicants and reviewers having a clear and aligned
understanding of which qualities are most important for an application to be considered as high impact.
• There was a concern about the removal of grades as a part of the review process. They commented that
although grades alone cannot predict success, it is one of the metrics that can be an indicator of the applicant’s
understanding in the biomedical field.
Criterion 2: Science and Scientific Resources & Criterion 3: Training Plan and Training Resources
Many societies/institutions were supportive of criterion 2 and 3 as they expected it would reduce the potential sponsor
and institutional bias and focus on the merit of the science, and quality of the training plan.
Several societies/institutions were supportive of the proposal to eliminate peer review of the sponsor’s funding and
resources for the proposed research as it was perceived as increasing equity and access for applicants from smaller/less-
funded institutions. Conversely, a few societies/institutions commented that the peer review of financial resources
should remain as one of the elements in review criteria.
NIH Center for Scientific Review
13 Sept 2023
5
Although many societies/institutions were supportive of removing “Sponsors, Collaborators, and Consultants” and
“Institutional Environment and Commitment to Training” from the current review criteria, several had concerns
regarding “Science and Scientific Resources” and “Training Plan and Training Resources” from the proposed review
criteria. Criticisms of the proposed plan include:
• Although “Science and Scientific Resources” and “Training Plan and Training Resources” are the revised
“Sponsors, Collaborators, and Consultants” and “Institutional Environment and Commitment to Training”, bias
would still remain. A few societies/institutions fear that bias would find its way back into the evaluation even
with the revised criteria.
• A few societies/institutions noted that a track record of success
should remain as one of the evaluative factors in the revised criteria.
Areas Requiring Further Clarification and Reviewer Training
Societies/institutions shared concerns about areas of the review criteria that they found confusing or challenging to
review and highlighted areas that might require enhanced reviewer training before implementation. Feedback includes:
• Enhanced training of the newly proposed review criteria is needed not only for reviewers but also for scientific
review officers, chairpersons, and applicants.
• Specific guidance and instructions are needed as it is unclear how to incorporate information from the
application materials into the evaluation appropriately.
• Concerns that criterion 2 and 3 should be clarified further to avoid
any duplicative evaluation as it seems to evaluate the similar components.
Comments on proposed changes in the application
The second major recommendation by the CSRAC WG was to revise the fellowship supplemental section of the PHS. The
guiding principles of the WG in making these changes were to (1) modify the application content to align with the review
criteria, (2) discourage the use of easy-to-assess but incomplete and often misleading indicators, (3) emphasize the
substantive statements pertinent to the individual student, and (4) shorten the application by restructuring the content
as word limited statements.
Many individual respondents as well as most scientific societies/institutions were in favor of revising the fellowship
supplemental section of the PHS and believed it would streamline the application, reduce burden, and improve clarity
and accessibility of the application process. Many respondents were in favor of the elimination of grades from this
section and applauded this decision as it enables reviewers to focus on better markers of potential success as a scientist,
in addition to increasing equity. A smaller subset of respondents believed that grades should still be evaluated, as they
believe past performance is often a predictor of future performance.
More detailed feedback from individuals and scientific societies/institutions on modifications to the fellowship
supplemental section of the PHS are presented below.
Comments from Individuals
Many respondents wrote in favor of the proposed revisions and believed they would streamline the application and
reduce burden. They also thought that these revisions will provide more structure and streamline the process.
A few respondents were critical of the proposed revisions to the application. Comments submitted include:
• The perception that the revisions are not adequate to diminish sponsor and institutional bias.
NIH Center for Scientific Review
13 Sept 2023
6
• The changes are positive but do not go far enough as the application still consists of many repetitive sections
across the applicant and sponsor/co-sponsor sections.
• Removal of undergraduate grades from consideration would hide a pattern of poor academic achievement,
which may indicate a potential weakness in the applicant’s knowledge.
• Burden has been added in the following areas:
o The proposed structure to reference letters would be an additional burden on letter writers and not all
the points to be covered are relevant.
o Revisions to the applicant section may increase burden for applicants by adding statements to write, as
well as for reviewers with additional statements to review.
Applicant Section
Many respondents wrote in favor of the proposed revisions to this section. There was enthusiasm among many for
eliminating grades from this section; respondents applauded this decision as it emphasizes achievement relative to
opportunities, as well as potential in addition to increasing equity.
A few respondents were unfavorable towards this section, criticisms of the proposed plan include:
• The view that the proposed review criteria changes will not lead to impartial review of applicants from labs and
institutions that receive less funding.
Sponsor and Co-sponsor Section
Some respondents wrote in favor of the proposed revisions to this section and believed that it eliminated emphasis on
sponsor’s/co-sponsors’ individual qualifications and made way for more junior faculty to serve as mentors.
A few respondents criticized parts of the proposed section. Criticisms include:
• Concerns that the quality of a sponsor and co-sponsor will be judged based on the number of trainees
supervised – this is seen as disadvantaging to newer mentors who have not been able to build up a lengthy track
record. Conversely, a few respondents thought that the track record of the mentor is an important component
to emphasize.
Letter of references
Many respondents were in favor of these revisions. Those in favor said the changes are welcome and will better allow
reviewers to assess the scientific potential of the applicant and their preparedness for the fellowship training.
Optional Statement of Special Circumstances
A few respondents wrote in favor of the addition of this section and believed it would be beneficial to those that might
have had personal/professional circumstances impact productivity.
Many respondents expressed criticisms of this section. Criticisms include:
• One concern was it could introduce bias into the process of peer review. Some respondents were concerned
that not all applicants would reveal circumstances due to personal beliefs, while others worried that not all
reviewers would judge all circumstances equally.
• An additional concern was that utilizing the Optional Statement of Special Circumstance will put undue influence
on the applicant to disclose personal information to the review panel, often leaders in the field.
Areas Requiring Further Clarification and Reviewer Training
NIH Center for Scientific Review
13 Sept 2023
7
Respondents shared concerns about areas of the Fellowship Supplemental Section of the PHS that are perceived as
confusing or challenging to review. Respondents highlighted areas that might require enhanced reviewer training and
clear instructions for applicants before implementation:
• The request for detailed guidance for applicants, sponsors/co-sponsors, and reviewers for how to both write and
review applications based on these new criteria.
• The need for detailed guidance on the information that is encompassed within each statement.
• A clear statement that junior faculty do not need a full co-mentor to support NRSA fellowship applications
would be beneficial.
• It would be beneficial to provide applicants with guidance for how to write training plans as well as to reviewers
for how to evaluate them.
• Detailed instructions on how to write the letters of reference are needed.
• It is not clear in how scores would be influenced by the Optional Statement of Special Circumstance section;
respondents would like more guidance for the review of this section and its impact on scores.
Comments from Scientific Societies and Academic Institutions
Many societies/institutions were in favor of revising the fellowship application materials and instruction, as it aligns well
with the revised review criteria and improves the clarity and accessibility of the application process.
While some societies/institutions acknowledged that the revised application materials would reduce the overall length
of the applications, a few societies/institutions noted that it would be necessary to simplify the application further and
decrease the workload of applicants, sponsors, and reviewers. A few societies/institutions noted that some of the
supplement sections lack clarity and asked for clarifications and further instructions throughout the applications. They
suggested having examples of successful fellowship applicants on the NIH website to guide applicants, sponsors, and
reviewers through the application materials and review process.
Application Section
Some societies/institutions supported the proposed changes to the “Applicant Section” as it addresses many issues
raised in the past, and they appreciated the elimination of undergraduate grades in the section. However, some
societies/institutions shared concerns about parts of the section. Criticisms of the proposed plan include:
• A few societies/institutions expressed that there may be redundancy among five elements in the Applicant
Statements section.
Sponsor and Co-sponsor Section
Some societies/institutions appreciated the proposed revisions to the “Sponsor and Co-Sponsor” section of the
Fellowship supplement as it would better emphasize the quality of the training by the sponsor instead of the quantity,
which would remove areas prone to bias.
A few societies/institutions shared concerns regarding the part of this section, including:
• That although quantity of the success does not equate to the quality of mentoring, it is still valuable to provide
evidence or outcomes of trainees in the past.
• That “Number of Fellows/Trainees to be Supervised During the Fellowship” might be used to penalize junior
sponsors if they currently have fewer trainees in their lab.
NIH Center for Scientific Review
13 Sept 2023
8
Letter of references
A majority of societies/institutions were supportive of the revised instructions for the “Letter of Reference.” However,
several of them provided a few recommendations:
• Having two letters of reference instead of three would be ideal as some applicants might face challenges to find
three appropriate references.
• Instructions should be further adjusted to welcome letters of reference from non-researcher contexts.
• Writing a letter creates burden for some people as it may be challenging to address all components adequately
about the applicant and suggested making it simple and easy for the writers.
Optional Statement of Special Circumstances
Many societies/institutions appreciated the proposed addition of the “Optional Statement of Special Circumstance” as
this would provide opportunities for applicants to contextualize the challenges that they may have faced in their
academic performances or career trajectory.
A few societies/institutions expressed concerns about this section. They suggested that applicants could invent
circumstances and reviewers would have no means to confirm the veracity of the statement.
Areas Requiring Further Clarification and Reviewer Training
Some societies/institutions expressed confusion around the “Applicant” section and “Sponsor and Co-Sponsor” section
in Fellowship Supplemental Section of the PHS. The feedback includes:
• Concerns regarding the new applicant’s self-assessment statement in relation to the sponsor’s evaluation on the
applicant.
o Some were unsure if the applicant and the sponsor should complete these sections on their own
without any influence from each other, or if they should discuss and complete these together. They
asked for the clarifications regarding these two sections.
o Some had concerns about potential coordination between the applicant and sponsor in writing the self-
assessment statement and sponsor’s evaluation.
• It is unclear how the Fellowship Qualifications and Self-assessment statements would provide different
information about the applicant.
Responsive Scientific Societies and Academic Institutions
Scientific Society
Academic Institutions
American Academy of Nursing
The American Association of Immunologists
American Physiological Society
American Society for Biochemistry and Molecular Biology
American Society for Pharmacology and Experimental
Therapeutics
Association of American Medical Colleges
Endocrine Society
Federation of American Societies for Experimental Biology
National Postdoctoral Association
The National Science Policy Network