How to (not) conduct a systematic review: the case of bevacizumab and ranibizumab

It was recently published in the International Journal of Ophthalmology a systematic review of the literature that compares the efficacy and safety of bevacizumab and ranibizumab in the treatment of age-related macular degeneration (AMD). Zhang and his colleagues collected data from 4 randomized clinical trials (RCTs) and 11 observational studies, for a total number of more than 4,000 patients. They concluded that the effectiveness of treatments is similar, but that ranibizumab is safer than bevacizumab . Regarding the events of ocular inflammation the authors found a relative risk (RR) of 0.45 with (95% CI 0.23 to 0.89), and for venous thrombosis a RR of 0.27 (95% CI 0.08 to 0.89). However this systematic review presents several methodological limitations.

Search of the literature not updated

The period of literature research is limited to October 2012, excluding from the analysis randomized clinical trials such as MANTA and GEFAL, and limiting the IVAN study data, a multicenter study in the UK for a period of two years, only to preliminary analysis of the first year of follow-up.

Critiques to the methodology of meta-analysis

The statistical analysis (the so-called meta-analysis) was conducted with arguable methodologies. This systematic review combines together the results from RCTs with those from observational studies. Some authors recommend to exclude the observational studies from meta-analysis, others advise to include them only for documented and valid reasons (eg the absence of RCTs, sample size or length of follow-up of the RCTs insufficient to observe the effect in study, etc): always the separate analysis of the two kinds of studies is strongly recommended. However, if the decision to combine RCTs and observational studies is taken, an appropriate and conservative methodology should be used (random-effects analysis), which produces confidence intervals wider than those produced with the fixed-effect analysis, in consequence of the greater heterogeneity present between the two types of study. In the case of death from any cause and of ocular inflammation the authors used the fixed-effect analysis, although they combined together observational studies with RCTs.

Lack of sensitivity analysis and of influence analysis

An analysis of the sensitivity and influence is lacking, the first to assess the robustness of the results with the choice of different methods of analysis (fixed-effect vs. random-effects) and the second to assess the influence of every single study on the overall estimate.

Reanalyzed data for ocular inflammation and venous thrombosis

As for the eye inflammation, re-analyzing data using the random-effects analysis we get a RR of 0.61 (95% CI 0.18 to 2.08), no longer statistically significant and clinically inconclusive. In addition, a sensitivity and influence analysis shows that the final results are strongly influenced by the Sharma study: this is an observational study, with estimates not adjusted for potential confounders, with patients with indications different than DMS and funded by Novartis. With the exclusion of this study we get a RR of 0.91 (95% CI 0.37 to 2.25), again not statistically nor clinically significant. For these reasons, it is conservatively to assume that the reduction in risk of ocular inflammation in favor of ranibizumab found in the review is at least unreliable.
Regarding the venous thrombosis, the result is due to the combination of two randomized clinical trials, CATT and IVAN, the second of which with incomplete data due to the search of the literature not updated. Reviewing the risk of venous thrombosis events using the updated data of IVAN, with 2 years follow-up, the RR of 0.43 (0.16 to 1.07) is obtained, not statistically significant and clinically inconclusive. It should also be remembered that the full meta-analysis should be recalculated for each event with inclusion of RCTs published in 2013.

The danger of the spread of systematic reviews of low quality

The shortcomings of this review induce the suspicious of a process of peer-review by the International Journal of Ophthalmology of low quality: its publication in a journal indexed in MEDLINE, however, makes it available to researchers around the world. For instance, this review is cited in the document signed by 16 experts and sent to the Italian Minister of Health Lorenzin together with a critical opinion on a document of the National Health Council (CSS). The CSS analyzed the problem of effectiveness and safety of bevacizumab and ranibizumab and concluded, on April 15, 2014, that the two drugs "do not show statistically significant differences in terms of efficacy and safety in the treatment of macular degeneration". On May 5, 2014, were also announced the preliminary results of a systematic review conducted by the Cochrane Collaboration, according to which there would be no differences in the safety profile between the two drugs in the treatment of DMS (RR 1.11 and 1.05, respectively, not statistically significant for mortality and serious adverse events). For the Cochrane review, in contrast to the one of Zhang, is available the detailed study protocol containing the list of primary and secondary outcomes.
A comprehensive summary of the whole story can be found in the May 2014 Focus.