Why an overhaul of randomized controlled trials concerning women’s health is in demand. A checklist to assess the integrity of research

ID: 

OS10.5

Session: 

Oral session: Innovative solutions to challenges of evidence production (2)

Date: 

Tuesday 22 October 2019 - 14:00 to 15:30

Location: 

All authors in correct order:

Mol BW1, Wertaschnigg D1, Gurrin L2
1 Department of Obstetrics and Gynaecology, Monash University, Australia
2 Centre for Epidemiology and Biostatistics, Melbourne School of Population and Global Health, The University of Melbourne, Australia
Presenting author and contact person

Presenting author:

Ben Willem Mol

Contact person:

Abstract text
Background: there are concerns about the quality and credibility of randomised clinical trials (RCTs) from various regions in the world. Current tools and procedures to prevent and detect scientific misconduct in clinical medicine are limited.

Objectives: to develop a checklist with tools to check the scientific integrity of RCTs.

Methods: we developed a checklist with toolbox to assess the integrity of published RCTs. Our checklist contained five domains:
1) presence of research ethics, documentation of placebo or other study medication, adequacy of trial registration;
2) feasibility of recruitment, feasibility of clinical data (do the data make sense?);
3) quality of statistical analysis (description of statistical methods, reproducibility of univariable statistical analysis);
4) possible inadequate data handling as indicated by the violation of Benford-Newcomb’s law for the distribution of the first leading digits;
5) Monte Carlo simulations and Kolmogorov-smirnov test to evaluate the probability of randomization.

For individual participant data, we propose four domains:
1) presence of original data such as date of birth or birthweight;
2) presence of normal distributions;
3) order of randomization;
4) excessive arithmetic progression and repeating numbers in continuous variables.

We systematically reviewed RCTs from one North-African country that were published in leading journals of Obstetrics and Gynaecology between 2013 and 2018, and applied the part of the checklist that assessed published papers.

Results: we identified 52 RCTs. The proportion of studies with proper registration was 35% (18/52). We could not reproduce sample size calculation in 44.4% (14/36) of studies. Regarding statistical analysis, 23.1% (12/52) of studies had critical concerns. The proportions of studies with not repeatable tests for continuous and categorical baseline variables were 68.8% (22/32) and 38.7% (12/31), respectively. The statistical result of the primary outcome was not repeatable in 29.4% (10/34) of studies. The distribution of first leading digits was found to violate the Benford-Newcomb’s law in 32.6% (15/46) of studies. With Monte Carlo simulations and Kolmogorov-smirnov test, the overall probability that studies were properly randomized as assessed from the baseline data across RCTs was 0.0009216.

With respect to individual participant data, we obtained data from two studies. The individual participant data indicated a high suspicion of fabricated data, with excessive arithmetic progression in both studies.

Conclusions: we report serious concerns about the overall quality of published RCTs in the field of women’s health from a North-African country. Our checklist to assess integrity can help to assess papers with fabricated data.