NICHD SMALL GRANTS PROGRAM

SHORT TERM EVALUATION SUMMARY

The NICHD Small Grants (R03) Program was introduced in the NIH Guide to Grants and Contracts on February 9, 1996 (PA-96-025). Until that announcement, NICHD had supported R03 grants only for a selected group of scholars participating in the Institute’s Physician Scientist (K12) programs.

The primary goals of the new NICHD Small Grants Program were to:

• Enable investigators to collect pilot or feasibility data to serve as the basis for future R01 applications, thereby increasing the likelihood of their success.
• Provide an opportunity for investigators to undertake a variety of small-scale projects with simplified application procedures and a relatively short time from submission to award.
• Enable new investigators, or those from institutions without well-developed research traditions or resources, to begin independent research activities.
• Increase the total number of Research Project Grants (RPGs) able to be supported by NICHD and decrease the average costs of RPGs.

As described in the Guide announcement, the NICHD R03 program demonstrates a number of specific, sometimes unique, features:

• The R03 grant provides up to $50,000 direct costs per year for one or two years and may not be renewed.
• Applications are accepted in all areas of science reflected in the Institute’s mission, but the specific types of small grants of particular interest to each branch also are published.
• Applications and awards are guided by Just-in-Time procedures.
• R03 grants may not be used to supplement ongoing projects or to support dissertation research.
• Only one revision of an R03 application may be submitted.

Based on data derived from applications submitted between 1996 and 1998, obtained from several NIH and NICHD databases, and from NICHD program, review, and grants management staff, an evaluation of the short-term outcomes of the NICHD Small Grants Program was undertaken. The goals of this evaluation are to:

• Help determine if the short-term goals of the program are being met.
• Inform Institute policy decisions regarding retention of the program in its current or modified form.
• Guide the development of a new Program Announcement for the next three years, if the program is to be retained.

An evaluation of longer-term goals (for example, improving the subsequent success of R01 applicants who have the opportunity to collect pilot data with an NICHD R03 grant) must await the availability of sufficient data. For example, only now are the first NICHD R03 grantees finishing the projects which may form the basis for a subsequent R01 application.

Between FY 1996 and FY 1998, NICHD received 478 applications in response to the Small Grants Program Announcement. The Institute awarded 123 R03 grants, at a total cost of $8.58 million.

Characteristics of the principal investigators and institutions applying for the NICHD R03 between 1996 and 1998 can be summarized as follows:

• Twenty-three (23) percent of the R03 applications were submitted by M.D.s, as compared to 19 percent of the NICHD R01 applications.
• The program attracted applications from a different group of institutions than those submitting the most R01 applications to NICHD. Three of the five institutions submitting the greatest number of R03 applications do not appear among the top five (or even 10) institutions for NICHD R01 submissions.
• Approximately 61 percent of NICHD R03 applications were submitted by new investigators, i.e., investigators who had never been PIs on NIH research grants. In contrast, approximately 32 percent of NICHD R01 applications were submitted by new investigators.
• Independently of the "new investigator" designation, the program attracted applications from all age groups, defined in terms of year of last degree.
• Approximately 29 percent of NICHD R03 applications proposed behavioral science research, compared to approximately 27 percent of NICHD R01 applications.

For the purpose of this evaluation, "success rate" is defined as the number of applications that are funded expressed as a percentage of the total number of applications submitted.

• The overall success rate for NICHD R03 applications is approximately 26 percent.
• This rate is slightly better than the overall success rate for NICHD R01 applications, which was 24 percent between FY 1996 and 1998.
• The relative success rates for biomedical and behavioral science R03 applications shows some differential, with the success rate for biomedical applications at 27 percent and that for behavioral applications at 22 percent. For NICHD R01 applications, the success rates for both types are 24 percent.

The following sections summarize the data relating to specific questions
addressed by the evaluation.

While new investigators demonstrate a lower success rate than established investigators for both R01s and R03s, the NICHD R03 program helps significantly to reduce the disparity between new and established investigators. With the termination of the First Independent Research Support and Transition (FIRST) award program (R29), new investigators will be dependent solely upon the R01, unless the R03 provides an alternative with a greater likelihood of success with a first grant.

• For both R01s and R03s, the success rate for new investigators is lower than that for established investigators. This differential applies both at the NICHD and across the NIH.
• The NICHD success rate for established investigators with R01s (29 percent) is almost twice that for new investigators (15 percent).
• For R03s, the differential in the success rate for established investigators (24 percent) compared to new investigators (20 percent) is much smaller.
• The success rate for NICHD R29s (new investigators only) was 23 percent.

With the introduction of the NICHD Small Grants Program, the K12 scholars were able to apply for the NICHD R03, with the increased funding it affords. For the purpose of comparison, two "classes" of K12 scholars were created: "Before Program" (n=18) are scholars who became eligible to apply for an R03 in the two years just prior to the start of the NICHD Small Grants Program. "After Program" (n=24) are scholars who became eligible to apply for an R03 in the two years following the start of the program.

• The application rate of the "After Program" group was 54 percent, compared to an application rate of 67 percent for the "Before Program" group.
• However, the success rate of the "After Program" K12 scholars increased to 69 percent from the 33 percent success rate of the "Before Program" group.
• "After Program" scholars were more likely to apply for and succeed in obtaining K08s than R03s.
• Seventy-five percent of all K12 scholars received their first awards from NICHD.

Since the introduction of the NICHD Small Grants Program, between FY 1996 and 1998, the total number of new research project grants (RPGs) awarded by NICHD has increased and the average total cost of a research project grant has decreased.

• Between 1996 and 1998, the number of new RPGs (R01, R03, and R29 only) increased from 154 to 255. Of the 101 additional awards, 68 were accounted for by R03s.
• Between 1996 and 1998, the average first-year total costs for a new NICHD RPG (R01, R03, and R29 only) decreased from $201,000 to $182,000. During the same period, the average first-year total costs for a new R01 increased from $228,000 to $262,000.

A unique feature of the NICHD Small Grants Program Announcement was the publication of the specific areas of interest of each branch with respect to the type of research (rather than area of science) it was particularly interested in supporting through the R03 mechanism. Because there was significant overlap in the interests of the branches, a total of nine separate types was identified (including "applications from new investigators"). A characterization of a large sample of applications submitted, however, suggests that these published interest areas are not necessarily the most popular for applicants.

• For only three of the nine published interest areas was there a match between the branch interest and the number of applications submitted, i.e., most of the interest areas failed to attract the most applications to the branches interested.
• Of the published areas, the three areas most frequently addressed included: (1) pilot/feasibility study; (2) innovative research; and (3) research methodology.
• However, the most popular type of research proposed in R03 applications can be characterized as "Underrepresented Topic or Approach." This area was not among those published. These applications appear to fill unique research needs or represent somewhat unusual approaches, outside the mainstream of science in a field.

From the first applications submitted to the NICHD Small Grants Program (for the October 1996 Council round) through the May 1998 Council round, all R03 applications were reviewed by the NICHD Division of Scientific Review (DSR). For the October 1998 Council round, the majority of new R03 applications were reviewed by study sections of the NIH Center for Scientific Review (CSR), while revisions were reviewed in DSR. For the January 1999 Council round, all NICHD R03 applications were reviewed in CSR. It is too early to draw conclusions about the relative success rates of applications reviewed in CSR as compared with DSR, although some initial observations may be made.

• There is great variability in success rates for NICHD R03s among study sections, whether in DSR or CSR. To some extent these rates may be meaningless, especially as they are affected by the absolute number of applications reviewed (e.g., if only one application is reviewed and funded, that is equivalent to a 100 percent success rate).
• The numbers of applications reviewed by a given DSR review group is generally larger than the number reviewed by most CSR study sections because applications are spread among many more CSR groups.
• The differential success rates experienced by the branches may be a reflection, in part, of the particular study sections to which their applications are primarily assigned.
• The most significant problem in the review of R03 applications, whether in DSR or CSR, appears to be the need to make reviewers aware of the differences between the R03 and R01, and to avoid application of unrealistic standards to the R03.

In 1997, NICHD program, review, and grants management staff, as well as the members of the NICHD R03 Secondary Review Committee, were invited to participate in focus groups. During these sessions, participants provided their perceptions of the program:

• Provides program staff with another tool to offer investigators when discussing funding opportunities.
• Increases funding options for investigators.
• Helps fulfill the NICHD mission.
• Provides a means to fill the gap left by the discontinuation of the R29 mechanism.
• Appears to be a good program for funding clinical research; these applications do not usually do well in study sections.
• Appears to benefit behavioral scientists, allowing them to conduct smaller studies.
• Ideal program for investigators who need to collect preliminary data for an R01.
• Tight payline impedes selecting innovative grants.
• Increases workload for DSR, GMB, and IRMB.

On the basis of the short-term evaluation of outcomes of the program thus far, as well as the feedback from Institute staff, it is recommended that the R03 program be continued, with several modifications:

• In place of branch-specific interest statements, Institute-wide interests and priorities should be published.
• The new Program Announcement should include features of the exploratory/developmental grant (R21), reflecting a growing interest among staff in this type of support.
• The new Program Announcement should emphasize the Institute’s interest in applications from new investigators, to increase even further the usefulness of the R03 mechanism as an alternative to the R01 for this group.
• Similarly, the Institute should consider including "new investigator" status as a specific criterion to be used to make funding decisions, in addition to percentile ranking alone.
• In general, the new Program Announcement should include a statement about NICHD funding policy for R03s.
• The new Program Announcement will specify Modular Grant Application, Review, and Award procedures. This is now required, and will replace and expand the Just-in-Time procedures currently in effect. These new procedures will simplify further the administrative requirements for both applicants and staff.
• With respect to locus of peer review, it is recommended that the success rates for NICHD R03 applications be analyzed when sufficient data become available. Attention should be given to potential differential success rates between DSR and CSR, among study sections, and/or among branches. On the basis of this analysis and other relevant factors, a decision about locus of review should be made.
• With respect to quality of peer review, SRAs must continue to be trained to assure that reviewers understand the difference between R01 and R03 review. The formal evaluation criteria for the two mechanisms are the same; we do not want to fund small studies of poor scientific merit. However, reviewers should recognize that R03 applications must be on a different scale, not of a different quality.
• Consideration should be given to a policy that would limit the award of R03 grants to investigators with other support totaling less than $500,000 in direct costs.

In several instances, issues regarding the small grant process within NICHD were resolved as they came to light.

• Initially, there was no time incentive for submitting an R03 application; R01s and R03s were being funded in the same time frame, post-Council. To resolve this, the Secondary Review Committee began to schedule its meeting approximately one month prior to Council and summary statements were required two weeks prior to the meeting. Funding decisions were then able to be made before Council.
• To further shorten the time between submission and funding, the Secondary Review Committee has resolved to apply expedited review procedures to applications falling within the payline, thus permitting a significant proportion of R03 applications to be funded shortly after initial review.