The evidence pyramid and its place in informed healthcare decisions: Comments, criticisms, and clarifications

I read with interest the article by Panigrahi and Bhattacharya¹ on reconsidering evidence synthesis. The goal of research (whether primary or secondary) is to find answers to scientific questions. Once this overarching concept is grasped by the scientific community, several other concepts immediately become clearer. Scientific questions can be answered by conducting either research, or search of the existing/available research. Thus, the goal of primary (sometimes feted as original) and secondary (sometimes vilified as desk/armchair) research is identical. Therefore, they are complementary and not competitive.

Viewed in this light, the evidence pyramid is not a ranking of different study designs, but stratification of the methods available (to answer questions), based on their ‘risk-of-bias’. Of course, it can be argued that the pyramid is most applicable to questions addressing the efficacy of healthcare interventions. Answers to such questions may be available through case reports, case series, case-control studies, cohort studies, trials, and randomized controlled trials (RCTs). The ‘risk-of-bias’ varies by the inherent nature of each study design (and methods used within a design to mitigate it). This is why RCT resides above all other primary research study designs, in the evidence pyramid. Obviously, RCT is an inappropriate design to address many important healthcare questions, such as associations between exposure(s) to risk factors and occurrence of events, prognostic factors in clinical conditions, epidemiological questions on prevalence or incidence, evaluation of diagnostic tests, etc. In such situations, it would be foolish to consider that non-RCT designs are inferior methods.

Third, properly designed and conducted systematic reviews (of existing research studies) are more efficient and economical to answer questions than ‘reinventing the wheel’ by undertaking primary research all over again. Further, systematic reviews are not simply collations or summaries of existing research studies, but also include critical appraisal of the included studies, to estimate the sources of bias within as well as across them, in order to evaluate the trustworthiness of the data. This is why they reside at the top of the evidence pyramid.

Fourth, systematic reviews are not synonymous with meta-analyses. The latter is simply a statistical method of pooling data from various studies. The analysis has advantages such as assigning weights to included studies, visualisation of the results (and weights) of included studies as well as the pooled effect, statistical modelling (based on various assumptions of the data), and interpretation of the pooled estimate based on statistical heterogeneity, and precision. Inappropriate statistical methods create erroneous conclusions in any research study. Meta-analyses are no different.

It cannot be emphasised enough that systematic reviews are not merely the integration or aggregation of studies (to pool data). They involve accessing all the relevant studies (that can answer the question ‘asked’ before initiating the systematic review), assimilating the relevant studies and extracting the data appropriately, appraising critically the data for each outcome and estimating confidence in the results, assessing the results in terms of the question, and only then considering applicability for policy or practice.² Appropriately conducted systematic reviews can identify the risk of bias associated with the results (and their interpretation), thereby increasing confidence. The GRADE approach³ goes beyond estimating the risk-of-bias, to consider diverse issues that can lower (or raise) confidence in the findings.^3,4

If primary research studies are conducted without due attention to issues that create high(er) risk-of-bias (such as inappropriate/deficient study design; participant sampling and enrolment methods; sample size; study procedures; outcome assessment tools and methods; participant follow up, tools of outcome measurement; data handling, analysis, and interpretation) their findings are untrustworthy. Naturally, incorporating such data into systematic reviews (and meta-analyses) would lead to untrustworthy results. But the beauty of systematic reviews is that the reviewers can choose to altogether exclude study designs with high(er) risk-of-bias, address the problem through sensitivity analysis, and even decline to pool data if inappropriate. These measures raise confidence in the findings of such reviews, cementing their position at the top of the evidence pyramid.

Just as there is a plethora of inappropriately conducted primary research, the problem exists for secondary research as well. Besides wastage of resources (human, financial, time), it creates two additional problems for end-users, viz. difficulty in making sense of the secondary research, and challenges in making decisions based on this. This has created a need for tertiary research,⁵ to assist end-users in making evidence-informed decisions.

Certain misperceptions¹ require clarification. From the foregoing, it is amply clear that the evidence pyramid is not a ‘teaching aid’. Second, systematic reviews occupy the top position not because they merely aggregate primary studies; but because they address scientific questions whilst also considering bias. Third, just as inappropriately conducted research studies muddy the waters for decision-makers, systematic reviews (based on them) have similar potential. This demands appropriate evidence synthesis, and not a pyramid excluding them. Fourth, the proposed two-triangle model is a simplistic depiction of various primary and secondary study designs, lacking emphasis on the purpose of either type of research.