Analysis of NIGMS Request for Information on Strategies for Modernizing Biomedical
Graduate Education
Kenneth D. Gibbs, Jr.
National Institute of General Medical Sciences
Office of Program Planning, Analysis, and Evaluation
Introduction
NIGMS has been actively involved in efforts to catalyze the modernization of biomedical
graduate education. These efforts have included hosting a symposium to showcase innovations
in biomedical graduate education and providing administrative supplements to T32 predoctoral
training grants to enhance rigor and reproducibility, career development and skills development.
On June 8, 2016, NIGMS released a Request for Information (RFI; NOT-GM-16-109) to obtain
input from the broader community on how to catalyze the modernization of biomedical graduate
education through NIGMS’ institutional predoctoral training grants program. The RFI, and an
accompanying Feedback Loop blog post, asked community members to provide input on the
following:
• Current strengths, weaknesses and challenges in graduate biomedical education.
• Changes that could enhance graduate education to ensure that scientists of tomorrow
have the skills, abilities and knowledge they need to advance biomedical research as
efficiently and effectively as possible.
• Major barriers to achieving these changes and potential strategies to overcome them.
• Key skills that graduate students should develop in order to become outstanding
biomedical scientists and the best approaches for developing those skills.
• Potential approaches to modernizing graduate education through the existing NIGMS
institutional predoctoral training grants.
• Anything else you feel is important for us to consider.
The RFI closed on August 5, 2016, yielding a total of 90 unique responses submitted through an
online form, the Feedback Loop blog and direct email to NIGMS staff members. Most of the
responses were anonymous, but the content indicated that comments were submitted by
students, faculty, institutions and professional societies. Stakeholder organizations, including
the Federation of American Societies for Experimental Biology, Association of American Medical
Colleges, Genetics Society of America, Future of Research, American Physiological Society and
Global Biological Standards Institute provided comments.
Analysis Approach
A team of NIGMS staff employed an iterative approach to establish themes and sub-themes
reflecting the content in the RFI responses. The first 30 responses were read to develop and
refine an initial list of codes. The team conducted two rounds of practice coding (seven to 10
responses each round) to ensure consistent understanding and application of the codes. In
each of these rounds, code definitions and application rules were refined, and codes were
added or deleted as appropriate. After receiving all responses, three readers independently
coded each response, and then the team met to discuss and resolve differences in code
application. The response formats were heterogeneous (i.e., some used the web form to
provide feedback, while others wrote letters), and many respondents reiterated their main points
in multiple sections of their response. Thus, codes were applied on a per response basis rather
than per prompt.
Results
Twenty-eight major themes were identified in the responses. These themes were grouped into
five major categories (each comprised of two to 15 themes): institutional and training-related
issues, skills development, systemic issues in the research enterprise, careers, and
administrative/review issues. Figure 1 shows how frequently these categories were represented
in the responses. Below are the codes and exemplar quotes that represent the breadth of
responses.
Figure 1: Major Categories in Biomedical Graduate Education RFI Responses. Bar chart
showing the number of RFI responses in which one of the major categories was represented. A
total of 90 unique responses were received for the RFI.
Category #1: Institutional and Training-Related Issues (97% of responses).
Fifteen themes comprised the “Institutional and Training-Related” category, and Figure 2 shows
the number of responses in which each code was present.
Figure 2: Institutional and Training-Related Themes in Biomedical Graduate Education RFI
Responses. Bar chart showing the number of RFI responses in which each theme that was part
of the Institutional and Training-related category was represented. A total of 90 unique
responses were received for the RFI.
The roles that principal investigators (PIs), and mentorship/advising play in graduate
education were prominent themes in this category. One respondent described the potential
positive and negative impacts that PIs and mentorship can play in the training experiences of
students:
“The greatest strength of US graduate education is the apprenticeship experience that is
provided by the best faculty advisors, i.e. those that have the best interests of the
student at heart. Having both advised many Ph.D. biomedical students, and then served
as a department and Graduate School leader at an [American Association of
Universities] institution, I have seen the very best and the very worst advising. When the
advising works, then the student is prepared for the career track that they have chosen,
and have the advisors support for whatever this should be. In many ways, I would argue
that a good advisor is very similar to a good parent. You “raise” them to succeed in life,
often in ways that you had not imagined.
The greatest weakness of US graduate education is the apprenticeship experience when
this results in an almost serflike relationship of the graduate student to the advisor –
especially common with foreign-nationality graduate students, and foreign-born faculty
advisors. I have seen too many exploitative relationships where students never leave the
lab, have few opportunities to learn and experience anything besides their lab research,
and are considered to be 100% “owned” by the advisors because they and not the
university is paying their stipend – even though the students are clearly on 40%
appointments. I have had faculty refuse to allow their students to take Preparing Future
courses and other ‘non-research’ activities that will help with the student’s future careers.
And while every advisor agrees that ethics is important, few address it in the lab, and
lead by example…”
In addition to the impact faculty members have on individual trainees, responses also
addressed the need to ensure that PIs have the skills necessary to excel in student training. For
example, managing large data sets is becoming increasingly important in biomedical science,
as is the ability to effectively mentor and advise trainees from increasingly diverse backgrounds.
However, responses indicated that not all PIs have the skills to excel in these areas.
“The faculty teaching these topics also do not have this inherent understanding of big
data, modeling and statistics. You cannot teach students new ways of thinking if you do
not understand and value the tools and approaches as the instructor.”
“Mentoring and management skills of the professoriate…may be limited, understandable
as they themselves have been trained almost exclusively in bench science.”
Curriculum and classroom teaching also represented a major area of attention. Many
comments focused on the need to ensure that modern pedagogical techniques are employed,
and that the curriculum addresses the range of skills (technical, operational and professional)
that biomedical trainees need to succeed. In response to the prompt asking what changes could
enhance graduate education, one respondent wrote:
“Provide funding for departments to hire educators/career counselors to improve the
graduate coursework quality (less didactic, more active learning) and teach non-
traditional course topics (writing, presenting, job search, interviewing, etc.).”
Others commented that current “graduate biomedical education practices are not generally
informed by research” on adult teaching-and-learning, and that “faculty members are busy, are
not incentivized to change their teaching, and are not themselves versed in evidence-based
active learning or online education.” Thus, responses indicated the need to update the
curriculum in graduate programs, as well as to support faculty to enhance their ability to
effectively teach and mentor trainees.
The resources required to conduct biomedical research and research training were also
discussed. Responses noted that facilities in the U.S. are high quality, and that there is
generally good financial support for students. However, a tough funding climate for principal
investigators may limit their ability to fully engage in training-related activities. As described in a
professional society’s response, “Increased competition for federal grant dollars and decreased
funding for state institutions have forced advisors to spend more time writing grants and less
time supervising their trainees.”
Issues of diversity and inclusion—especially with respect to race, ethnicity and gender—and
the role that institutional climate plays in shaping student experiences were also prominent.
As one respondent noted, “Efforts to modernize and enhance graduate education must take into
account the environments in which training occurs and the relationships which impact the
training experience.” All of the responses that mentioned diversity and inclusion (except two)
indicated support for NIGMS’ efforts in these areas, and encouraged additional efforts:
“Gender and racial diversity remain a significant problem in science. Deeper
commitment to graduate training is an opportunity to address some of these disparities,
and I urge to consider how they can be interwoven into the reforms you enact.”
“Lack of diversity continues to be an alarming problem in biomedical research. Given our
changing demographics, this is no longer a “minority problem,” but rather a national
emergency.”
“Diversity should not be considered as an add-on to graduate education, but rather, a
vital and necessary element to endure innovation and scientific excellence. It should be
incorporated into all academic training protocols (perhaps even ethics trainings) so that
students understand that to be an excellent scientist, diversity is a professional element
where they must gain knowledge and work to master. Working to eliminate bias and
preference in the lab is vital for the biomedical workforce to move forward in a
progressive and authentic way.”
There was also strong support for interdisciplinary training in Ph.D. programs. As described
by one of the institutional responses, “In addition to discipline-specific knowledge that is needed
in order to train the next generation of scientists, it is becoming increasingly important that
trainees have exposure to areas that extend beyond their traditional area of inquiry.” Another
response noted, “Science is no longer a single discipline act anymore. Innovative students need
to be able to reach into other areas of research and combine ideas in new ways.” Comments
also emphasized that student training should prepare them to conduct rigorous and
reproducible research; work effectively in team science environments; and translate their
basic research discoveries into therapies, products or intervention approaches that directly
impact patients or the public.
Other themes in the institutional and training-related category included:
• Flexibility. These comments encouraged NIGMS to allow flexibility in the use of training
grant funds “to integrate new modes of preparing trainees,” and focused on ensuring that
Ph.D. programs have flexibility so that students can tailor their course of study to their
professional goals.
• Time-to-degree. These comments were split between those encouraging efforts to make it
shorter (“The time to graduate is too long”), and those who felt too strong a focus on
shortening time to degree may adversely impact efforts to enhance career preparation
during Ph.D. training.
• Stipends. Many comments argued that student stipends should be increased (either across
the board or adjusted for cost of living), while a few recommended eliminating stipends
altogether.
o Master’s degrees. These responses pointed to the need to consider the role that master’s
degrees play with respect to modernizing biomedical graduate education (either as a
prerequisite to, or an alternative for Ph.D. training).
o Dual degrees. These responses focused on issues specific to those in M.D.-Ph.D. training
programs, or proposed the creation of other dual degree programs (such as Ph.D./M.B.A. or
Ph.Ds. combined with other master’s degrees).
Category #2: Skills Development (91% of responses)
Consistent with the RFI prompts, many of the responses focused on skills. Figure 3 shows the
frequency with which major skills categories were described.
Figure 3: Skills Development Themes in Biomedical Graduate Education RFI Responses. Bar
chart showing the number of RFI responses in which each theme that was part of the skills
development category was represented. A total of 90 unique responses were received for the
RFI.
There was broad agreement that professional and transferrable skills (e.g., communication,
teamwork, time management) are important to learn no matter what career path a trainee
pursues. In response to a prompt asking the key skills graduate students should develop to
become outstanding biomedical research scientists, one respondent wrote,
“Communication. Time management. Quality writing. These are all skills that will be
crucial to long-term graduate student success, whether they stay in the academy or seek
out private industry careers.”
Another response noted that within interdisciplinary research teams, there is “increasing value
on effective communication, interpersonal skills, and time management” and that “intentional
training in these skills can enhance the effectiveness of scientists across disciplines and career
paths.”
Many responses also emphasized the importance of ensuring that trainees continue to develop
strong scientific and research skills (e.g., experimental design, critical thinking/analysis). As
one response stated:
“To become an independent scientist requires three essential skills – the ability to think
critically and to write and speak effectively about science. I would argue that developing
and honing these three skills in the context of research-based training remain at the
heart of effective graduate education.”
Many of the responses expressed the sentiment that the current system does a good job at
developing these skills in trainees. In response to the prompt asking current strengths in
graduate biomedical education, one respondent said, “Graduate training programs remain a
cornerstone of developing rigorous thinking and the development/execution of scientific
research projects,” while another commented, “We are training students in bench skills,
generating hypotheses, designing studies, and analyzing data, such that our trainees have
much expertise in their field of study and are quite competent scientists.”
Quantitative and computational skills (e.g., statistical analysis, manipulating large data sets,
computer programming) were mentioned in over half of the responses. These responses were
nearly uniform in indicating the need for “increased focus/training in quantitative/computational
biology (rigorous data analysis, computational methods, bioinformatics, how to handle big data)”
no matter one’s discipline. At the same time, the responses indicated the need to integrate
these new quantitative skills with a “good understanding of the underlying science of their
system,” so that that students can “ask good questions and…design a study that effectively
addresses that question.”
Category #3: Systemic Issues in the Research Enterprise (70% of responses)
Systemic issues within the research enterprise, and their impact on graduate education, were
also mentioned in a majority of the responses. Figure 4 shows the relative frequency with which
each theme in this category was represented in the responses.
Figure 4: Systemic Issues in Research Enterprise Themes in Biomedical Graduate Education
RFI Responses. Bar chart showing the number of RFI responses in which each theme that was
part of the systemic issues in the research enterprise category was represented. A total of 90
unique responses were received for the RFI.
Specifically, the culture and incentives of science (i.e., what is or is perceived to be rewarded
by NIH and academic institutions; n=46) and incentive conflicts (i.e., when the best interests
of the faculty member and student are not totally aligned; n=23) were discussed in many of the
responses. Quotes from three separate responses below describe this issue:
“The training model for predoctoral students may present inherent conflict where the
student is considered both a trainee and a worker. As a trainee, it is recognized that
predoctoral students are still gaining experiences and competencies that are necessary
for their development as independent scientists. A major part of this development is the
creation of new knowledge and presenting this knowledge in the context of what is
known. However, trainees are often tasked with producing a product that is required for
the continued funding or advancement of the mentor. If handled incorrectly, this can
result in a conflict between the time and resources required for training, be it in discipline
content, communication skills, or in expertise not immediately linked to the project at
hand, and the completion of a “work product” for the mentor…”
“Predoctoral biomedical researchers have a recognized dual role as both trainees and
employees (NIH NOT-OD-15-008) and in particular the National Institutes of Health have
clarified the requirement to “support the development of skills critical to pursue careers
as independent investigators or other related careers”. In practice, training is a
secondary priority to bench science: bench science productivity is incentivized by
rewarding data generation with publications and grants, essential currencies in the
research enterprise. Training outcomes, however, are not highly valued in graduate
biomedical research except in the production of a dissertation (itself possibly a collection
of publications).”
“Our junior faculty in particular are under tremendous pressure to publish and bring in
funding, and they rely heavily on their doctoral students to help them meet expectations.
Sometimes the focus on research can get in the way of their support for the student's full
educational experience. Anything NIGMS can do to relieve the pressure on junior faculty
and support them in developing their research careers will therefore also improve the
educational experience of their doctoral students. The most prominent example of the
tension between faculty pressures and student educational needs is the reluctance of
advisors to support their students in professional development opportunities that
broaden their skills and perspectives.”
In addition to issues around incentives, responses also described how the sources of Ph.D.
student funding (i.e., training grant or individual fellowship versus support on an investigator’s
research grants) impact student experiences. Particularly, the comments indicated that when
students are supported by an investigator’s grant, their main role may be viewed as labor
(coded under the provision of labor theme), which can in turn negatively impact their
educational experience:
“The major barrier to radically changing the traditional graduate education model is the
ownership that individual faculty advisors have over trainees. Faculty rely on PhD
trainees to produce data, papers, etc. needed to secure more grant funding, obtain
tenure for junior faculty, etc. To overcome this, trainees should not be “owned” by an
individual faculty advisor, but rather they should have more autonomy and be aligned
only within a department or within a college or the graduate school. Trainees would
obviously need to work within an individual lab to complete their dissertation work, but
that lab PI should not see them as their own “employee.” Creating more autonomy would
give trainees more freedom to look after themselves in preparing for whatever career
they want to pursue.”
Although not part of the RFI prompts, 25 responses referenced postdoctoral scientists
(postdocs). Most these responses pointed to the fact that many of the systemic issues that
impact graduate students also address postdocs (with the phrase “students and postdocs” used
in many of these responses).
Category #4: Careers (60% of responses)
Career development also featured prominently in the responses, as shown in Figure 5.
Figure 5: Career-Related Themes in Biomedical Graduate Education RFI Responses. Bar chart
showing the number of RFI responses in which each theme that was part of the career related
issues was represented. A total of 90 unique responses were received for the RFI.
Specifically, the responses described the need for enhanced career development, especially
the current career landscape in which very few trainees attain faculty careers. While most
trainees go on to have careers outside of the professoriate, the responses expressed the
sentiment that, “graduate school is really not effective preparation for jobs beyond academia,”
and that the current system “fails to meet the needs of trainees who are not likely to pursue an
academic career.” Multiple responses indicated that “it is still largely taboo to openly prepare for
careers outside of the academic track,” and that “many faculty [members] don't know how to
advise students to be anything else but PIs.” Responses did concede there has been some
movement in the direction of career development (noting the NIH
Broadening Experiences in
Science Training (BEST) program), but pointed out much more can be done:
“Although important measures have been taken to prepare students for careers outside
the academic track, more must be done. There must be time made available for potential
internships and exposure to industry, consulting, data science, science writing, policy,
etc. so that students will have the requisite experience in nonacademic fields to be
competitive.”
Category #5: Administrative and Review (34% of responses)
Figure 6 shows major administrative and review-related issues present in the RFI responses.
Figure 6: Administrative and Review-Related Themes in Graduate Education RFI Responses.
Bar chart showing the number of RFI responses in which each theme that was part of the
administrative and review category was represented. A total of 90 unique, responses were
received for the RFI.
Peer review of training grants was discussed in 17 responses. For example, respondents
encouraged NIGMS and other institutes to use a broader suite of metrics when determining
what constitutes successful training:
“T32 training programs use the number of first-author student papers, time to degree,
and whether students go on to academic careers as metrics of success, making it
challenging to shift a culture toward embracing broader career options for students or
creating additional courses or activities that would support student professional
development. We strongly encourage NIGMS to look into the issue of what constitutes
success of training as measured by peer review of T32 renewal applications.” (emphasis
original)
“[Professional society] supports holistic review of training programs as related to their
impact on trainees, institutions, and society, and educating reviewers to define success
more broadly than pursuing an academic research career. During evaluation of grant
applications, training programs should not be criticized for participation or utilization in
institutional training efforts. Rather, programs should be encouraged to partner with
existing opportunities on their own campuses. Reviewers should also allow for some
variance in time to degree in review of T32s (as noted above, training should be based
on competencies, not based on time). [Professional society] also urges NIGMS
incorporate the expectation that institutions have a built-in evaluation and dissemination
plan within the T32 application, to ensure that best practices and outcomes are widely
and rigorously shared with the community.”
Beyond the criteria for review, responses also suggested enhancing the diversity of the
reviewers themselves. A group of university leaders noted that “training grant study
sections…are largely populated with people who have spent their lives in universities,” and they
urged “NIH to include a significant number of PhD scientists and engineers from industry,
government laboratories, and nonprofits on the panels that provide critique and rating for
training grants, in order to incorporate reality-grounded feedback on how to broaden career
perspectives effectively.”
Accountability and oversight was also discussed in several responses. Students wrote about
the need for university oversight to ensure faculty members’ relationships with their trainees
remain professional and promote students’/trainees’ development:
“There is essentially no accountability when a PI is abusive or highly inappropriate, and
going to the administration or causing problems at HR would ONLY serve to make a
student's life even worse while not correcting the situation.”
“My graduate program…did not have any oversight for students…My advisor often just
wanted me to do work that was totally unrelated to my thesis. There needs to be some
kind of system in every graduate program that keeps abusive (or maybe inexperienced --
my advisor had like 2 trainees before me) advisors in check.”
These responses included the need for more uniform standards both across and within
programs in a manner that allows flexibility, but also enhances quality of training for all students:
Graduate biomedical education differs when compared to many fields in that there is
essentially no regulation other than agencies involved in general program accreditation.
As such, no uniform competencies are required for students in biomedical research
fields. While regulation might add considerable burden for programs and program
administrators, experience in other graduate level training programs (e.g., the MD
degree) indicates that regulations can lead to greater rigor, higher and more uniform
standards, and greater institutional resources devoted to achieving those standards.
Imagine the world of hospital administration, patient care, residency or medical student
education without standards.
Finally, these responses also indicated the need for programs to widely disseminate the
outcomes of training investments. In response to the prompt asking about potential approaches
that would ensure that trainees have the skills and knowledge needed to enter the biomedical
research workforce, one respondent wrote:
“Track comprehensive, up-to-date metrics on graduate programs (e.g. graduation rate,
duration of program, job placement rate and fields). Make this data broadly available and
easily visualized. As scientists, we should know that before everything, we must have
data. With observations, patterns may manifest themselves and we can measure impact
of interventions.”
The issue of administrative burden was mentioned in only four responses.
Innovative Ideas
In addition to these categories, the analysis team looked for comments that proposed innovative
ideas that have not been raised during NIGMS’ earlier efforts to obtain input. For example, one
respondent suggested adding experimental aims to the T32 program to drive innovation:
“Establishing a clear expectation that training programs fund the innovative development
and implementation of approaches to address specific training challenges can recast
T32’s as experimental tools. Perhaps the inclusion of an experimental “aim” in T32s that
applies a specific intervention against a training challenge, with an appropriate plan of
assessment and refinement, should be accepted as a requirement for this granting
mechanism. Overall, the use and acceptance of training grants and training grant
supplements to drive innovation should be expanded.”
Another respondent suggested that NIGMS give broader consideration “to who can serve as PI”
of any new training programs, saying, “It should not be just R-level holding faculty. Career
advisors should also be allowed to be PI,” with the idea that doing so would allow programs to
better build upon the BEST models.
Conclusion
NIGMS received a diverse set of responses to its Request for Information for strategies for
modernizing biomedical graduate education. These responses covered institutional and training-
related issues, skills development, systemic issues within the research enterprise, career
development, and administrative/review issues. NIGMS recognizes that those who responded to
the RFI are unlikely to represent a random subset of the entire stakeholder community impacted
by graduate biomedical education. However, these responses provide insights regarding how
members of the extramural community view the current challenges and opportunities in
graduate biomedical education, and will inform NIGMS’ ongoing efforts to catalyze its
modernization.
Acknowledgement
Austin Oh, Christa Reynolds and Kris Willis contributed analysis to this report.
Graduate Education RFI Themes
Institutional and Training-Related Issues
1. Principal investigators (aka, advisors or mentors)
• Definition: References the impact of PIs/advisors/mentors in student training.
2. Mentorship/advising
• Definition: References to the advising or mentorship relationship (positive or
negative).
3. Curriculum and classroom teaching
• Definition: References to coursework (content and structure for delivery), and
classroom teaching in graduate programs.
4. Resources
• Definition: References to the physical (e.g., buildings, equipment, etc.) and financial
(e.g., funding), and other resources (e.g., time) necessary to conduct biomedical
research.
5. Diversity and inclusion
• Definition: References to race/ethnicity or gender with respect to training
environment or career development.
6. Institutional climate
• Definition: the social, cultural aspects of a university training environment (either
positive or negative). Includes mentions of student wellness/mental health.
7. Interdisciplinarity
• Definition: References to training students to think across disciplines (either
favorable or those encouraging greater focus within single disciplines).
8. Rigor and reproducibility
• Definition: References to efforts to improve scientific rigor and reproducibility.
9. Team science
• Definition: References to team science, and/or the need for scientists to be able to
work successfully in multidisciplinary teams.
10. Translation
a. Definition: References to the development or commercialization of basic research
discoveries into therapies, products or approaches that directly impact the public
(e.g., patients, public health).
11. Flexibility
• Definition: References to the need for flexibility, personalization, tailoring,
customization, etc., in Ph.D. programs that allow trainees to better meet their career
and professional needs.
12. Time-to-degree
• Definition: reference to the time it takes to complete a Ph.D. program.
13. Master’s degrees
• Definition: References to the role that master’s degrees play in graduate education.
14. Stipend
• Definition: References to the stipends Ph.D. students.
15. Dual degree
• Definition: References to combined degree programs (e.g., M.D.-Ph.D., or other
Ph.D. combined degrees).
Skills
16. Professional & transferrable skills
• Definition: References to the skills (and the development of those skills) that are
important in many professional settings such as communication, teamwork,
management, finance, etc.
17. Scientific & research skills
• Definition: References to the skills needed to be a capable research scientist such as
experimental design, critical/analytic thinking, etc.
18. Quantitative & computational skills
• Definition: References to quantitative skills and/or computational biology. Examples
include computer programming, data science, statistical analysis, etc.
Systemic Issues
19. Culture and incentives of science
• Definitions: References to broader issues and reward/incentive structures within
biomedical science and academic institutions. For example, references to what is (or
is perceived to be) rewarded by NIH and institutions.
20. Sources of Ph.D. student funding
• Definition: References to the funding source of Ph.D. students (e.g., training grant
and fellowships versus a PI’s research grant). Also, includes discussion of restriction
of NRSA awards to citizens.
21. Postdocs
• Definition: References to postdoctoral scientists and their roles within the research
enterprise.
22. Incentive conflict
• Definition: References to the conflicting incentives for PIs/advisors and their trainees.
23. Provision of labor
• Definition: refers to students and/or postdocs as a source of (cheap) labor for PIs;
includes discussion of abuse, slavery or ownership, may overlap with discussion of
conflicting incentives.
Careers
24. Career landscape
• Definition: References the current career prospects for biomedical Ph.Ds. Often
coupled with the fact that most Ph.Ds. do not go on to pursue faculty careers, or
references to the number of Ph.Ds. trained relative to the number of faculty positions.
25. Career development
• Definition: References to training Ph.D. students for a breadth of careers, including
careers beyond academia (e.g., industry, “alternative careers,” science writing,
science policy, etc.). Also includes specific references to internships, or other work-
based manners to gain career relevant skills during Ph.D. training.
Administrative and Review
26. Peer review
• Definition: References to the peer review or training and research grants.
27. Accountability/oversight
• Definition: References to (i) oversight by funding agencies of grantees, or of faculty
members at universities, or (ii) accountability for grantees to follow through on the
terms of their grants and disseminate their outcomes.
28. Administrative burden
• Definition: References to the number of requirements for the administration of federal
training grants.
