Core GRADE 3: rating certainty of evidence—assessing inconsistency

When Core GRADE users construct PICO frameworks that are broad with respect to both patients and interventions-as we believe they should-they must prepare for the possibility of inconsistent results. They do so by identifying a priori hypotheses to explain inconsistency, including a postulated direction. Having decided on their subgroup hypotheses, Core GRADE users address the key criteria for evaluating inconsistency. Examining the forest plot, they note the magnitude of differences in point estimates, the extent to which the CIs overlap, and where the point estimates lie in relation to the target of their certainty rating. The greater the variability in point estimates and the less the overlap of CIs, the more likely there is problematic inconsistency. The decision, however, requires consideration of the chosen threshold for certainty rating: whether the null or the MID, the greater the extent to which, in the presence of minimally overlapping CIs, point estimates fall on opposite sides of the threshold, the more likely there is problematic inconsistency. Problematic inconsistency requires determining if a priori hypotheses can explain that inconsistency. Critical criteria for judging the credibility of any apparent subgroup effects include whether the analysis is based on within trial or between trial comparisons, the P value of a test of interaction, and whether the analysis is based on a small number of a priori hypotheses with a specified direction. If the subgroup effect proves credible, Core GRADE users will provide separate evidence summaries for each subgroup and rate certainty of evidence accordingly. If not, they will assess inconsistency across all eligible studies.