Translate this page into:
Publication-Equivalent as the new single currency of research impact: The ICMR-Impact of Research and Innovation Scale (ICMR-IRIS)
secy-dg@icmr.gov.in
-
Received: ,
Accepted: ,
Research and innovation are key drivers of human development. As the apex body for biomedical research in the country, the Indian Council of Medical Research (ICMR) provides numerous research grants every year. In 2024-25, ICMR provided research grants amounting to over 2100 crore rupees. It is critical to evaluate the impact of research and innovation grants and activities to maintain accountability in the use of public funds. Presently, project reports of individual completed projects are evaluated by experts for satisfactoriness before the project is considered closed, and a narrative annual report summarizes outputs from research projects supported by ICMR. A simple yet effective tool for research and innovation impact evaluation has remained elusive.
Research impact is often considered to be a demonstrable contribution that research makes to society. Biomedical research could contribute to furthering science, improvements in clinical care or in public health programmes, and the development of innovative health technologies, including diagnostics, therapeutics, vaccines and devices. A useful impact assessment tool should evaluate all the above-mentioned contributions of research and innovation.
Currently available tools for research impact evaluation: Quest for a perfect tool, scale or index has thus far proven elusive. Several measures were proposed in different contexts, and a few survived the test of time. Currently available methods are summarized below.
Research impact of an individual researcher or an institute is often assessed by h-index or i10-index1. The h-index represents the number of publications that have been cited at least that many times, whereas the i10-index represents the number of publications that have been cited at least 10 times. These indices, based on citation of one's work by others, at best, are only able to measure the scientific impact of research.
Almetrics Attention Score measures the amount of digital attention an article receives through multiple sources2. News, blogs, online encyclopaedias pages, policy documents, patents, and circulation in social media are used to assign a score.
Translational Research Impact Scale (TRIS) is a comprehensive tool using a set of 72 indicators organized into three broad research impact domains and nine subdomains3. The large number of indices and complexity of the scale hinder its wider use in routine practice.
Based on the foundational work in an earlier round by NITI-Aayog, Office of Principal Scientific Adviser to Government of India recently published a report of the second round of review on innovation excellence in 244 public-funded institutes. This review reports data on over 60 indicators across three broad areas for review (organizational effectiveness, science-technology and innovation excellence and socio-economic impact)4.
Thus, we have a spectrum of methods ranging from those based only on the number and citations of publications to those that are based on a large number of indicators. The former are easy to use but only reflect scientific impact. The latter cover multiple domains of impact but are cumbersome to use and do not give a single measure of impact comparable across research activities.
The idea of Publication-Equivalent
The basis of the new ICMR-Impact of Research and Innovation Scale (ICMR-IRIS) is the idea of converting all domains of research impact to a single measure, the Publication-Equivalent (PE). Publications in peer-reviewed indexed journals are the fundamental output of primary research projects or systematic review and meta-analysis. One such publication is therefore considered to be equal to one PE. If a research publication resulted in a change of clinical guidelines or public health programmes, it is assigned an additional 10 PE. In simple terms, if a research publication had an impact on either clinical care or public health practice, it is valued as having published 10 research papers.
Innovations often follow the route of patents, technology development and its commercialization. In the ICMR-IRIS, one granted patent is given a value of five PE. Similarly, all other indicators of impact are converted to a single measure, summarized as the PE.
ICMR - Impact of Research and Innovation Scale (ICMR-IRIS)
ICMR-IRIS is a simple and inclusive measure of research and innovation impact, in which a new index, PE is used for assessing the impact of research on multiple domains. The values of PE for different types of impact have been assigned after extensive discussions with the senior management team of ICMR and some external experts. The PE values assigned to different domains of research impact are summarized in the table. Total PE for a research grant, a researcher or an institute is calculated by adding the PE scores for all eight indicators (Table). It should be noted that multiple researchers and institutes collaborating on a project would each get a PE in their evaluation, because the idea is to measure contribution and not attribution. Further, this approach is likely to encourage and enhance collaboration. Some practical considerations for the application of ICMR-IRIS indicators are discussed below:
| Indicator | Publication-Equivalents (PE) | |
|---|---|---|
| Research publication | ||
| A publication in indexed peer reviewed journal reporting results or methods of primary research, or a systematic review & meta-analysis | 1 | |
| High scientific impact | ||
| A research publication that has received 100 or more citations, or is published in previous 5 yr in a journal with an Impact Factor of 10 or more | 2 | |
| Clinical care impact | ||
| A research publication that has resulted in a documented change in national or international clinical care guidelines | 10 | |
| Public health impact | ||
| A research publication that has resulted in a documented change in State, national or international public health programme guidelines | 10 | |
| Patents | ||
| A national or international granted patent | 5 | |
| Health technology developed and used in limited settings | ||
| A new health technology that has been used in limited settings, such as research or surveillance | 10 | |
| Health technology developed and commercialized | ||
| A new health technology that has been commercialized & used at scale | 20 | |
| Preclinical or clinical evaluation of health technologies developed by others | ||
| A preclinical or clinical evaluation that resulted in regulatory approval or public procurement of the health technology | 3 | |
Research publications
A publication reporting results or methods of primary research, or reporting a systematic review and meta-analysis, is given a value of one PE. Other types of publications, such as case reports, editorials, opinion pieces, perspectives, narrative reviews, etc., are not considered.
High scientific impact
The impact of research in furthering science is indicated by citations received by them. In ICMR-IRIS, a research publication that has received 100 or more citations is considered to have substantial scientific impact and is given an additional score of two PEs. Publication in a journal with an impact factor of 10 or more is also alternatively given a similar score if the paper has been published in the previous five years, because getting 100 citations may take time.
Clinical care impact
If a research publication results in documented change of clinical guidelines and thus has an impact on clinical care, it is assigned an additional value of 10 PE. The documentation should be in the form of a statement within the guideline, a reference to the publication, or as an acknowledgement.
Public health impact
If a research publication results in documented change of a public health programme and thus has an impact on public health practice, it is assigned an additional value of 10 PE. The documentation should be in the form of a statement within the guideline, a reference to the publication, or as an acknowledgement.
Patents
Innovations are often patented and not put in public domain through publications in peer-reviewed journals. ICMR-IRIS assigns a value of 5 PE for each granted patent. However, no PE are assigned for filing a patent.
Health technology developed and used in limited settings
An innovation that results in the development of a health technology that is used in limited settings, such as for research or specialized surveillance, earns a score of 10 PE.
Health technology developed and commercialized for extensive use
An innovation that results in the development of a health technology that is commercialized and used at scale earns a score of 20 PE. Use at scale refers to public procurement or availability for purchase in the open market.
Clinical or preclinical evaluation of technology developed by others
Translational research includes studies to evaluate health technologies developed by industry or others. This refers to validation of a diagnostic with clinical samples, a preclinical regulatory study, a clinical trial, etc. Such an evaluation or validation study that is either used for regulatory approval or for public procurement is given a value of 3 PE in ICMR-IRIS.
First use cases for ICMR-IRIS
The ICMR-IRIS is being used for the evaluation of ICMR intramural and extramural research grants programmes. Research impact of each of the ICMR institutes is being calculated by assessing their performance of the previous five years (2020 to 2024) across the eight indicators of ICMR-IRIS. Directors of the Institutes were asked to provide the data on indicators with clear documentation, which was verified by the Director General's office. ICMR-IRIS was used to assign a single measure of their impact, as the total PEs for five years. Inputs of five years with a time lag, as it takes 3-5 years for research grants to have an impact (2016 to 2020), in terms of total research funds received and the total number of scientist years are also being collected. The final performance measures for an institute are PE per year, PE per crore rupees, and PE per scientist-year. These performance measures allow comparison of different institutes, as well as the aggregated research impact of the ICMR intramural programme. A similar exercise has been completed for the extramural research grants programme of ICMR, and the results of these two exercises will be published soon.
The ICMR-IRIS scale has also been piloted on a single research project, as well as the lifetime research impact of an individual researcher. Expected PE using ICMR-IRIS has also been piloted to design possibly more impactful research studies. Finally, the expected PE has been tried as a measure to select proposals for research funding by ICMR. In the above cases, users have provided favourable and encouraging feedback.
To conclude, ICMR-IRIS is a scale developed by ICMR to evaluate the impact of research and innovation. The scale is flexible; it can be used to calculate the research impact of a project, a researcher, an institute, or a funding agency. It is simple to use and is based on easily verifiable data. The scale results in a single measure, total PE, which can be further standardised by the number of total projects, grant amount, or number of scientist-years. The principles of ICMR-IRIS can be used for research areas other than biomedical and health research by modifying the indicators and values assigned to them. It may be possible to develop an automated artificial intelligence-based system to implement ICMR-IRIS for faster and efficient research and innovation impact evaluations. Preliminary experience suggests that use of ICMR-IRIS has the potential not only of evaluating research impact in a wholesome manner, but also to encourage more impactful research in the future.
Financial support & sponsorship
None.
Conflicts of Interest
None.
Use of Artificial Intelligence (AI)-Assisted Technology for manuscript preparation
The authors confirm that there was no use of AI-assisted technology for assisting in the writing of the manuscript and no images were manipulated using AI.
Reference
- An index to quantify an individual's scientific research output. Proc Natl Acad Sci USA. 2005;102:16569-72.
- [CrossRef] [PubMed] [Google Scholar]
- A critical review on altmetrics: Can we measure the social impact factor? Insights Imaging. 2021;12:92.
- [CrossRef] [PubMed] [PubMed Central] [Google Scholar]
- The translational research impact scale: Development, construct validity, and reliability testing. Eval Health Prof. 2014;37:50-7.
- [CrossRef] [PubMed] [PubMed Central] [Google Scholar]
- Evaluation of Innovation Excellence Indicators of Public Funded R&D Organizations' (Round 2). Available from: https://www.psa.gov.in/article/evaluation-innovation-excellence-indicators-public-funded-rd-organizations-round/9389, accessed on June 29, 2025