CIHR Reviewers' Guide for the Canada Postdoctoral Research Award (CPRA) program

Table of Contents

Introduction

On behalf of CIHR, we would like to thank reviewers for agreeing to serve as peer review committee members. The success of the peer review process is made possible by dedicated people like you who generously give their time and expertise. Your efforts are greatly appreciated by CIHR and the scientific community.

The purpose of this document is to provide information on the policies, procedures and process for the peer review of applications submitted to the Canada Postdoctoral Research Award (CPRA) program.

Note: As part of reviewers' engagement in peer review at CIHR, we ask reviewers to take a moment to complete or update the Reviewer Profile on ResearchNet. The Reviewer Profile has been developed as a tool to build and support reviewer expertise management. For the Canada Postdoctoral Research Award (CPRA) program, it is important to complete the First Independent Appointment section.

Peer Review at CIHR

Information on CIHR's objectives, governance and policies; an outline of the roles and responsibilities of peer review committee members; and the policies, principles and procedures for peer review of applications can be found in the CIHR Peer Review Guide for Training and Salary Awards. It is important that reviewers become familiar with this document, as well as the present document, before starting their reviews.

Summary of the Peer Review Process

The Canada Postdoctoral Research Award (CPRA) program uses an structured review process, using the online ResearchNet platform. The review process is completed in one stage: an individual written review and rating of an assigned set of applications (i.e. there is no committee meeting). All eligible applications will be assigned to three (3) reviewers.

There are two (2) peer review committees for the CPRA program. Each application will be assigned to the committee with the scientific area that has been chosen by the applicant, which most closely aligns with the applicant's proposed research activities.

For more information regarding the committee mandates refer to: Peer Review Committees and Mandates – Canada Postdoctoral Research Award (CPRA) program.

Reviewers are asked to follow the step-by-step instructions below to successfully complete all peer review tasks:

  • Step 1: Read the pertinent documentation
  • Step 2: Manage Conflicts/Ability to Review
  • Step 3: Conduct Reviews
  • Step 4: Submit written reviews and ratings on ResearchNet
  • Step 5: Be prepared for a re-review

Step-by-Step Instructions

Step 1: Read the pertinent documentation

The peer review process for this funding opportunity is described in detail in this document. It is essential to read the document and be familiar with it. It is also important to complete or update your Reviewer Profile on ResearchNet, complete the Reviewing for the Canada Postdoctoral Research Award (CPRA) program eLearning modules and read the following:

Step 2: Manage Conflicts/Ability to Review

To identify conflicts of interests on the assigned set of applications, reviewers are to follow these steps:

  • Log into ResearchNet.
  • On the home page, click on the link of their assigned committee to open the main task list.
  • Complete the task "Review Confidentiality Policy and Conflict of Interest Guidelines" (once completed, it will "open" the other tasks).
  • Open the "Manage Conflicts/Ability to Review" task.
    • For each assigned application, use the information provided to indicate if you are able to review or if there is a conflict. If there is a conflict, CIHR will reassign the application to another reviewer.

Notes:

  1. CIHR assigns CPRA applications to one of two peer review committees based on the primary scientific area identified by the applicant. The committees are multidisciplinary and applications are not assigned to reviewers based on specific expertise. As such you are likely to be assigned applications that are not within your specific area of expertise. Reviewers are asked to assess applications from a generalist's perspective and should not declare conflicts simply because applications are outside their area of expertise. Although you may not be familiar with the field, we ask that each reviewer complete their reviews from the point of view of a generalist.
  2. As other reviewers within your committee will also declare conflicts, it is likely that you will receive additional application assignments. The calibration of workloads will be maintained to ensure a fair peer review workload.

Step 3: Conduct Reviews

Once conflicts have been identified, the full content of remaining assigned applications will be available under the task "Conduct Reviews". Reviewers should then follow the steps below.

3.1 Review the selection criteria

Reviewers should first become familiar with the selection criteria found at the end of this document in Appendix A. This appendix provides an interpretation of each criterion and identifies which elements of the application to review for that criterion.

CIHR is a signatory of the San Francisco Declaration on Research Assessment (DORA), which recognizes that scholarly outputs include a broad range of contributions and are not limited to published journal articles. Reviewers should therefore consider:

  • A range of contributions (e.g., research publications, reports, books, guidelines, datasets, code, tools, training and mentorship, volunteerism, community engagement, standards, software and commercialized products) and impacts (e.g., influence on policy and practice, influence on direction of thought, use of research results by stakeholders, health outcomes, societal outcomes, and distinctions-based, meaningful and culturally safe health research); and
  • The context of the applicant (e.g., health problems, family responsibilities, disabilities, trauma and/or loss, pandemic impact, career stage, area(s) of research, experiential knowledge and diverse career paths) and how it may have impacted the applicant's performance (see range of contributions). Please see specific notes about research area and career stage:
    • Reviewers should consider the research area/discipline of the applicant when reviewing an application, for example, publication productivity can vary when comparing a biomedical researcher, a clinician or a social scientist.
    • Reviewers should consider the career stage of the applicants to better assess and calibrate their set of applications, for example, direct entry to fellowship from PhD vs. entry after several years outside academia. Reviewers are asked to think critically about whether the training position for which an individual is applying for funding will have the desired career benefits and impact compared to other applicants.

Reviewers should guard against placing too much value on the number of contributions – the focus must be on the impact and significance of the contributions. Reviewers should not use journal-based metrics such as the Journal Impact Factor as surrogate measures of quality and/or excellence as they introduce bias into the review process. Citation rates vary between disciplines and contexts; members must be mindful of this when considering them as part of their evaluation. Metrics such as number of publications and citations, and size and number of grants/awards should not be used in isolation to assess the applicant's achievements, contributions, and activities.

As stated in DORA, the "scientific content of a paper is much more important than publication metrics or the identity of the journal in which it was published". Applicants are instructed to elaborate and comment on their role, positive impact and importance of their research contributions for their 3 most significant contributions.

3.2 Read the assigned applications

Reviewers should read all their assigned applications in detail before rating any of them and take notes to capture their impressions. The Canada Postdoctoral Research Award Reviewer Worksheet in Appendix B provides a template that they may wish to use. This worksheet is for reviewers' own personal use and will not be filed with CIHR.

It is important to note that many applicants will likely be conducting research outside of the reviewer's research specialty. Reviewers should therefore review each application with a generalist's perspective and assess the overall quality of the application including the applicant's career goals, profile and expectations as well as the proposed research, using the appropriate selection criteria.

It is important that reviewers are aware of the importance of the integration of sex and gender in research proposals and assess their appropriateness where applicable in their assigned set of applications. To learn more about Sex and Gender considerations in Research, reviewers are encouraged to complete these training modules:

When evaluating applications that involve and engage with First Nations, Inuit and Métis Peoples, reviewers are strongly encouraged to review the following training module and literature:

As the agencies have signed the San Francisco Declaration on Research Assessment (DORA), a global initiative whose purpose is to support the development and promotion of best practices in the assessment of scholarly research. DORA recognizes the need to improve the ways in which research is evaluated, beyond widely used journal-based metrics. As signatories of DORA, CIHR recognize and value a broad range of contributions and emphasize their quality and impact. Reviewers are strongly encouraged to review the list of resources available to them.

To ensure that all applications are treated equally, reviewers are asked to base their evaluation only on the content of the application and not to complete any additional research (e.g. publications via PubMed, etc.).

3.3 Learning for participants in peer review

CIHR offers a number of learning modules to help you gain in-depth knowledge about our programs, processes and tools. These modules are intended to ensure that all participants in the peer review process have the same base knowledge of the processes and policies in order to conduct effective and fair peer review.

Addressing bias

As reviewers read the applications, they should be aware of and take actions to mitigate against bias related to gender, ethnicity, disability, Indigenous identity and Indigenous ways of knowing, institution size, region, age, language (including official language and minority communities bias) and interdisciplinary approaches to research. For example, it is important to be aware that:

  • Career interruptions for childbearing and family responsibilities can influence opportunity for knowledge production, publications and related variables;
  • Different disciplines, environments and diverse career paths offer different opportunities for research contributions (e.g., publications, influence on policy and practice, patents, knowledge mobilization activities, etc.);
  • The reputation of institutions should not affect the reviewer's view of applicants or their research training environment;
  • It is important to take steps to mitigate bias in reviewers' thought process about differences in culture (e.g. Indigenous Peoples of Canada);
  • Recognize Indigenous identity in a reconciliation, nation-to-nation framework; and,
  • A point should be made of respecting Canada's linguistic duality by recognizing the value of research in French and the value of research on Francophone minority communities.

To learn more about bias in peer review, reviewers must complete the related learning module.

Sex and Gender in Health Research

Reviewers must complete one of the Sex and Gender in Health Research online modules self-selected by each reviewer to align with their area of expertise:

3.4 Rate their assigned applications

Reviewers are asked to score their assigned applications against each of the selection criteria described in Appendix A, using CIHR's rating scale (below). It is particularly important that the full scale be used.

Applications with an overall weighted score below 3.5 are not eligible for CIHR funding, including those from partnerships/priority announcement opportunities.

Descriptor Range Definition Outcome
Outstanding 4.5 – 4.9 The application excels in most or all relevant aspects. Any short-comings are minimal. May be Funded
Excellent 4.0 – 4.4 The application excels in many relevant aspects, and reasonably addresses all others. Certain improvements are possible.
Very Good 3.5 – 3.9 The application excels in some relevant aspects, and reasonably addresses all others. Some improvements are necessary.
Good 3.0 – 3.4 The application broadly addresses relevant aspects. Major revisions are required. Not Fundable
Average 2.0 – 2.9 The application fails to provide convincing information and/or has serious inherent flaws or gaps.
Below Average 1.0 – 1.9
Not Acceptable 0.0 – 0.9

3.5 Provide a written assessment for each assigned application

Reviewers must provide a concisely written assessment (limited to 4000 characters) for each assigned application that supports their ratings, highlighting the strengths and weaknesses of the application for each selection criteria. The written reviews will provide constructive advice to applicants to assist them in improving the quality and efficiency of the proposed training.

Reviewers should focus their comments on the strengths and weaknesses for each selection criterion:

  • Keep it simple;
  • Use familiar descriptors, such as those from the CIHR rating scale, that align with your rating;
  • Include justification, context and an explanation of your comments, if applicable, for each topic introduced;
  • Respect DORA principles by considering a range of contributions and impacts and avoiding the use of journal-based metrics such as Journal Impact Factors as surrogate measures of quality. Do not use metrics such as number of publications and citations, and size and number of grants or awards in isolation to assess the applicant – instead, balance quantitative metrics with qualitative parts of the proposal;
  • Be clear and concise;
  • While brevity is acceptable (e.g. using bullets), express complete thoughts and ensure the length is sufficient enough to inform the reader;
  • Use objective and non-inflammatory language – avoid any biased language or approach;
  • Carefully avoid language that might be construed as sarcastic, flippant, arrogant or inappropriate in any way.
  • When referring to the applicant use gender neutral pronouns or phrases. For example, use "they" or "the applicant," rather than "he" or "she".

The applicant will receive the review as it is submitted by the reviewer. For this reason, reviewers are to refrain from inserting scores in the written comments and should not identify themselves in order to ensure the confidentiality of the review process.

For additional guidance on conducting high quality reviews please refer to CIHR's Review Quality Expectations and Review Quality Checklist, as well as the Conducting Quality Reviews learning module.

3.6 Flag issues for CIHR's attention

Any concerns regarding the application, eligibility, ethics, human stem cells, etc. should be reported to CIHR staff immediately for follow-up and should not be noted in the written comments unless they impact the scientific quality of the application.

For the full list of potential issues, please refer to the CIHR Peer Review Guide for Training and Salary Awards.

Similarly, if you believe there may be a misrepresentation of information in an application or a breach of Agency policy, please inform us immediately so we can determine whether the issue needs to be addressed through the responsible conduct of research (RCR) process. For a list of potential breaches, please refer to the Tri-Agency Framework: Responsible Conduct of Research.

Step 4: Submit written reviews and ratings on ResearchNet

For security reasons, ResearchNet times-out rather frequently, and CIHR is unable to retrieve any lost information. Therefore, reviewers should save their work often so that it is not lost. Alternatively, reviewers can write their comments offline using the template as proposed in Appendix B and then copy and paste the text directly into the appropriate text box on ResearchNet when ready.

Also, as the reviewers perform their evaluation, the reviews can be saved as drafts by selecting "Save draft copy" on ResearchNet. This allows reviewers to make changes at a later time. However, in order to submit the written reviews and ratings to CIHR, reviewers must select "Submit Final Review". Afterwards, no modifications will be possible.

It is important for reviewers to respect the deadline provided by submitting their written reviews and scores via ResearchNet by the date specified via correspondence with CIHR staff responsible for this program. Delays in the peer review process will jeopardize CIHR's ability to release decisions to applicants by the published date. If, at any point in the process, a reviewer determines that they may not be able to submit their reviews on or before the deadline, they must contact CIHR staff as soon as possible.

Step 5: Be prepared for a re-review

Once all the overall weighted scores are submitted, CIHR performs a discrepancy review by calculating the final rating for each application. CIHR will then identify applications which are at risk of an unfair decision because of a wide spread between the reviewers' overall score. In such cases, CIHR will ask the assigned reviewers to reconsider their initial assessment and resubmit scores. To do so, they will be asked to get in contact with each other to discuss the application. Reviewers do not have access to the other reviewer's scores/reviews. However, when necessary, CIHR staff responsible for the program will help facilitate the re-review process.

For this purpose, it is recommended that reviewers keep their working notes on file until a few weeks after the competition results have been announced. After which, please delete and/or destroy any copies of the applications that you have printed, downloaded, or saved.

Please view the Guide on Handling Documents Used in Peer Review for instructions on the proper handling of peer review documents.

Ranking of applications

Upon completion of the peer review process, CIHR will generate a ranking list for the applications. Ties between overall scores are often found within these competition-ranking lists. CIHR has a process in place should the number of ties found in the fundable range exceed the overall budget allocated for the competition. The ties are broken using the committee percent rank for ties between applications in different committees, and the consolidated percent rank for ties between applications within a committee.

Feedback

An important component of the peer review process is the review of the committee's effectiveness and functioning, and feedback on policy issues that may have arisen during the process. This feedback provides an opportunity for staff to address any concerns from committee members and for staff to record comments on the peer review process as part of CIHR's ongoing efforts to maintain an effective and high-quality peer review system. Reviewer feedback should be communicated to the committee program officer by email.

Appendix A – Canada Postdoctoral Research Award (CPRA) Selection Criteria

CIHR has signed the San Francisco Declaration on Research Assessment (DORA), which recognizes that scholarly outputs include a broad range of contributions and impacts (e.g., influence on policy and practice, societal outcomes, distinctions-based, meaningful and culturally safe research) and are not limited to published journal articles.

Guard against placing too much value on the number of contributions; focus must be on the impact and significance of the contributions. Reviewers should not use journal-based metrics such as the Journal Impact Factor as surrogate measures of quality and/or excellence as they introduce bias into the review process. As stated in DORA, the "scientific content of a paper is much more important than publication metrics or the identity of the journal in which it was published".

It is important that reviewers take into account the context of the applicant (e.g., health problems, family responsibilities, disabilities, trauma and/or loss, pandemic impact, area(s) of research, experiential knowledge and diverse career paths) and how it may have impacted the applicant's performance.

Reviewers should take into account the career stage of the applicants to better assess and calibrate their set of applications (e.g., direct entry to fellowship from PhD vs. entry after several years outside academia). Also note that longer time spent conducting postdoctoral work (e.g., more than 5 years) or post-health professional training may not necessarily result in a higher probability of obtaining an academic position.

There are two selection criteria for the Canada Postdoctoral Research Award (CPRA) program, they are as follows.

1. Research Potential and experience (50%)

Note: In ResearchNet, the "Research potential and experience" criterion will be labelled as "Research Ability or Potential".

Criterion Notes/advice to reviewers Where to look:

a) Honours, Awards and Academic Distinctions

Assess official recognitions (i.e. distinctions, Honours and Prizes/Awards) received by the applicant, including training awards (competitive or not, monetary or not, declined … etc.)

  • Assess the importance and breadth of the applicant's official recognitions and special distinctions relative to their education, training and work experience.
  • Note any indications of special academic distinctions received.
  • Determine relevance to research and whether the recognition is regional, national or international.
 See Applicant's CCV including sections 'Recognitions' and 'Research Funding History'

b) Research-Related Contributions and Activities

Reviewers should consider a range of contributions (e.g., research publications, reports, books, guidelines, datasets, code, tools, training and mentorship, volunteerism, community engagement, standards, software and commercialized products) and impacts (e.g., influence on policy and practice, influence on direction of thought, use of research results by stakeholders, health outcomes, societal outcomes, and distinctions-based, meaningful and culturally safe health research). Look for:

  • Evidence of impactful research contributions relative to opportunities to date. Consider the significance, relevance and impact of their work.
    • Bear in mind that research dissemination practices and opportunities to publish may vary according to research discipline and life course (e.g., health professional career, time spent raising children, etc.).
    • Relate the publication dates to the applicant's education and training. Consider the list of abstracts as an indication of conference presentation activities.
    • Note: Do not consider publications listed with a "Submitted" status. Metrics such as number of publications and citations should not be used in isolation to assess the applicant's performance. Avoid using journal-based metrics, such as Journal Impact Factors, as a surrogate measure of the quality of individual research publications. As stated in DORA, the "scientific content of a paper is much more important than publication metrics or the identity of the journal in which it was published".
  • The applicant's role in research-related contributions and activities, and their estimated percent contribution to the work.
  • Research funding entries whereby the applicant was formally recorded as co-applicant, co-investigator, co-knowledge user, collaborator, decision maker, policy maker, principal applicant, principal investigator or principal knowledge user.
    • Note: some applicants enter training awards (e.g. CGS-Master's and CIHR Doctoral awards) in this section but they should be assessed as part of the "Honours, Awards and Academic Distinctions" criterion.
  • The applicant's other professional activities.
 See Applicant's CCV including sections Activities', 'Contributions' and 'Most Significant Contributions' and 'Special Circumstances' documents

c) Sponsor's Assessment of the Applicant's Characteristics and Abilities

Review the sponsor assessments of the applicant. These assessments should come from individuals under whom the applicant has trained, who are familiar with the applicant's characteristics and abilities and/or who have had a good opportunity to assess their potential for research. Keep in mind that applicants have no opportunity within the application to provide a justification for their choices of sponsors. In the Sponsor Assessment Forms:

  • Look for evidence from the sponsors that the applicant exhibits the characteristics and skills that correlate with research career achievement
  • Examine the sponsor's assessments, recognizing that positive comments are common while negative ones are not.
  • Read the supporting text carefully, taking note of the extent to which the sponsors justify their scores.
  • Look particularly for indications that the sponsors perceive the applicant to be an investigative type, that is, someone whose thinking is critical, questioning, original and independent.
  • Look for indications that the sponsors perceive the applicant to be both energetic and capable of being highly focused.
  • If the applicant has had an opportunity to conduct research, look for mention of creativity in setting research goals, designing experiments, developing new methodologies, interpreting findings and presenting results in writing.
  • Consider the following:
    • Do the detailed comments support the ratings outlined on the first page of the assessment?
    • How long has the sponsor known the applicant?
    • What is the relationship of the sponsor to the applicant?
 See the sponsor forms that provide an assessment of the applicant

2. Quality of proposed research program (50%)

Note: In ResearchNet, the "Quality of proposed research program" criterion will be labelled as "Proposed Research Project".

Criterion Notes/advice to reviewers Where to look:

a) Training Expectations

Consider how the applicant's previous training relates to the present proposal and elaborates on career goals.

  • Assess the clarity and logic of the applicant's plans for a research career and the relevance of the proposed training.
  • Consider how the training they expect to acquire will contribute to their research goals and impact they hope to achieve, and how this award will enable them to establish themselves as independent investigators.
    • Applicants have been asked to describe how their previous training (academic or otherwise) will benefit their current/upcoming postdoctoral/post-health professional training work.
    • Applicants have been asked to justify the need for further time spent in a postdoctoral/post-health professional training position (if relevant) and how it will contribute to their career goals.
    • Applicants have been encouraged to consider through careful career planning to assess their career aspirations, establish career objectives and to identify strategies to develop research, technical and professional skills in order to attain those objectives.
  • Consider any justification of why they chose the proposed training location and what they expect to learn from the training experience.
  • If applicable, consider any justification if they are planning to hold this award in the same research environment, and/or with the same supervisor as where they completed their doctorate degree. (i.e., research institution or its affiliate). Note: applicants should not be penalized when the choice of research environment is duly justified (Examples of valid justifications are scientific reasons (e.g., availability of specialized equipment/infrastructure or expertise), family reasons (e.g.,family obligations), health reasons (e.g., proximity to health care facilities), applicants conducting research with Indigenous communities and/or reasons related to community or cultural responsibilities. Indigenous communities are broadly defined as individuals, groups, organizations, and populations who self-identify as Indigenous (First Nations, Inuk (Inuit), andor Métis) living and working anywhere in Canada.).

See Training Expectation section

b) Proposed Research Project

Assess the research project summary, which applicants should be complete in collaboration with the proposed supervisor(s) and which should be written in general scientific language.

  • Determine if the proposed project is adequate to the applicant given their education, experience and interests. Evaluate if the project is the right balance of challenge, importance of the research question and feasibility in relation to the applicant's experience and training. Assess the integration of sex and/or gender in their research proposal where applicable. Note: it is not the project per se that is being assessed. The project should be viewed as an integral part of the applicant's development as a researcher.
  • The proposed research project summary should:
    • Include the specific hypothesis of the research and describe the applicant's role on the project;
    • Provide a concise account of the subject matter, an overview of each part of the research plan, specific project aims and the methodology;
    • Reflect the significance of the project.

See Research Project Summary document

c) Research Training Environment

Consider how elements of the research environment will contribute directly or indirectly to the quality of the applicant's research training experience. Consider whether there is a demonstrated commitment of the proposed supervisor(s) and the institution to support the development of the applicant's research project (funding, facilities, equipment, etc.) and professional development.

  • Determine if the research environment, including space, facilities, funding, collaborations, and personnel support available is appropriate.
  • Determine if the supervisor demonstrates a commitment to develop and/or facilitate the development of the trainee's research, technical and professional skills and networks.

See Research Environment document

Appendix B – Canada Postdoctoral Research Awards (CPRA) Reviewer Worksheet

The following table is meant to guide reviewers in the evaluation of the application. It is strictly for their working notes and will not be filed with CIHR.

Applicant Name: ___________________________________ Application #: __________________

Criterion Score Reviewers Comments

1. Research potential and experience (50%)

Note: In ResearchNet, the "Research potential and experience" criterion will be labelled as "Research Ability or Potential".

 a) Honours, Awards and Academic Distinctions  

Strengths:

  •  
  •  

Weaknesses:

  •  
  •  
b) Research-Related Contributions and Activities  
c) Sponsor's Assessment of the Applicant's Characteristics and Abilities  

2. Quality of proposed research program (50%)

Note: In ResearchNet, the "Quality of proposed research program" criterion will be labelled as "Proposed Research Project".

 a) Training Expectations  

Strengths:

  •  
  •  

Weaknesses:

  •  
  •  
b) Proposed Research Project  
c) Research Training Environment  
Date modified: