Background: systematic reviews of intervention effects may draw conclusions based on effect direction rather than statistical pooling of effect estimates. The Cochrane Handbook of Systematic Reviews of Interventions, 2018 (Handbook) now offers guidance on alternative methods for synthesis without meta-analysis. This guidance includes approaches to vote counting based on effect direction, and use of a sign test to determine whether there is any statistically significant evidence of an effect based on effect direction across studies.
Objective: to apply the new Handbook guidance on alternative synthesis methods based on effect direction and the use of sign tests.
Methods: we re-analysed data for three key outcomes from a published Cochrane Review, which exclusively used synthesis of effect direction and in which multiple similar outcomes from single studies were incorporated. We removed reference to statistical significance from the format of the effect direction plot and from the algorithm for overall within-study effect direction. We used a sign test to examine the probability of observing the given pattern of positive effect direction across studies if the null hypothesis of even distribution of positive and negative results were true. We compared the conclusions from this synthesis to the conclusions of the original review.
Results: the revised effect direction plot (Figure 1) presents results from a synthesis of 10 studies based on the revised algorithm. Removal of statistical significance from the algorithm to determine an overall effect direction for multiple similar outcomes within a single study did not change the assessment of effect direction. The sign test suggests that positive trends in effect direction across studies are statistically significant for housing condition (P = 0.0039), but not for general health or respiratory health (P = 0.2188). These results are consistent with the conclusions of the original review, which stated that housing improvements can lead to improvements in health but that the evidence is inconclusive. We could not include studies in which the effect direction was conflicting or unclear in the sign test, which reduced the power of the test.
Conclusions: the modified effect direction plot with the addition of a sign test is a feasible method of synthesising the best available evidence when meta-analysis is not possible and may improve transparency for reviews relying on effect direction. However, use of the sign test for small groups of studies may be problematic due to lack of power. Furthermore, the sign test does not consider study size, and risks overlooking evidence where the effect direction is unclear. The use of a sign test to assess trends in effect direction across studies demonstrates that vote counting based on effect direction should be interpreted with caution when the number of studies is small and/or effect direction in some studies is inconsistent.
Patient/consumer involvement: none (application of statistical methods)