To the editor:
We read with great interest the article by Glujovsky et al. (1), followed by the editorial comment of Dr. Legro (2), evaluating the level of evidence published in five infertility journals with a high impact factor. Authors concluded that both systematic reviews with meta-analysis (SRs) and randomized controlled trials (RCTs) were published at a disappointingly low level of 1.5 and 6.8%, respectively, with 15% of them multicenter RCTs. More than 90% of all the publications in the top five journals were neither SRs nor RCTs.
Our thought is on the mixture of positive and negative impact of the current finding.
The positive is that of RCTs, 90% were confirmed as real RCTs, demonstrating a low risk of bias and indicating good quality control by the journals through appropriate checklists and statements, even though only a quarter of the trials were registered in clinicaltrials.gov. The application of strict criteria has increased the quality and validity of the reports, leading to more valid and robust results, which are the necessary blocks toward building the right recommendations in health care.
The negative is that top-quality evidence is still today at a smaller than expected level; high costs and the length of time needed to conduct the studies constitute the major reasons for this. We do not think that this will change quickly in the near future. Adequately powered RCTs and their potential SRs usually require thorough sample size calculations, large numbers of participants, and pre-specified ways of allocation concealment and randomization; as a result, their implementation is often extremely difficult. The lack of appropriate funding, majorly justified in the current economic crisis, aggravates the problem.
Regarding published outcome measures, we would like to share the speculation raised by the editor on what is the outcome of an infertility trial (2); we agree and further question: why be “a developmentally normal 3-year-old infant” and not “a middle- or advanced-aged woman with no cancer in her personal history; or another one with three normally conceived healthy children,” and so on?
Further thought concentrates on a more realistic assessment of what the ultimate target, the quality of recommendations for health care providers, researchers, editorial board members, and especially patients, will resemble in the future. We firmly believe that the hierarchy of evidence will not and should not change (3); we also think that maybe the validity of the studies not belonging to these two top categories might rise in terms of their value of participating in the formation of medical policies. Apart from the fact that “Randomized controlled trials are not the solution to every problem in clinical medicine” (2), this might be a time for change. First, the universal adoption of primary and secondary outcomes of each intervention (at least in the infertility world) should be a basic step onward. Further steps might include other types of studies that are more easily conducted but retain an adequate power of evidence, or other studies that contain a combination of quantitative and qualitative data, along with simpler and more nonexperienced researchers, friendly types, and modes of analyses.
Charalampos Siristatidis, M.D., Ph.D.
Assisted Reproduction Unit, 3rd Department of Obstetrics and Gynecology, Attikon Hospital
University of Athens, Athens, Greece
Charalampos Chrelias, M.D., Ph.D.
3rd Department of Obstetrics and Gynecology, Attikon Hospital, University of Athens
1. Glujovsky M, Riestra B, Coscia A, Boggino C, Comande D, Ciapponi A. Assessment of research quality in major infertility journals. Fertil Steril 2012; 98:1539–43.
2. Legro R. Quo vadis randomized controlled trials in infertility? Fertil Steril 2012;98:1350–1.
3. Haynes RB. Of studies, syntheses, synopses, summaries, and systems: the ‘‘5S’’ evolution of information services for evidence-based health care decisions. ACP J Club 2006;145:A8.
Published online in Fertility and Sterility doi:10.1016/j.fertnstert.2013.01.118
The authors respond:
It is encouraging to see that a paper about the quality of research had repercussions both on the editorial board and among readers. The more importance journals give to this issue, the higher probability of getting better quality studies in the future.
We agree with most concepts written by Dr. Siristatidis and Dr. Chrelias in their letter. The current scene is neither so dark nor so bright. First, we agree that journals did a good job to help MEDLINE correctly index the real RCTs, given the high proportion of properly indexed studies. However, it is not equal to say that risk of bias is low, as this classification depends on several domains that should be assessed (which, in this case, most studies coincidentally showed low risk of bias).
Second, we think that there is still a heavy workload to be done in order to get more randomized controlled trials and systematic reviews registered, given that the registration is associated with better quality studies. Currently, the main source of essential protocol information can be found in trial registries, as defined by the World Health Organization (WHO) standards (1). In the same way, the international prospective register of systematic reviews (PROSPERO) is a promising new initiative, the purpose of which is to stimulate collaboration, reduce duplication of efforts, and improve the quality of systematic reviews (2,3). Editorial boards have been working on this in recent years and should keep doing it.
Third, we are concerned about the possibility that the real direct or indirect funding rate is higher than that reported in our paper. This issue should be further evaluated by journals when accepting a manuscript, re-assessing the tools used to recover this information.
Fourth, we also agree with Dr. Legro and Dr. Siristatidis about the need to consistently report more important outcomes. Maybe the confection of statements to establish standards of suggested primary outcomes might help to have more homogeneous results that allow comparison among different publications. That statement could be written by a team, with members coming from the editorial boards of journals, scientific societies, and independent researchers.
Last, we agree on the issue that the proportion of top quality design studies is probably not going to rise in the near future and, therefore, we should think about a strategy to make it happen. In addition, we could work on giving lower quality study designs a standardized tool to evaluate their quality and to raise the bar in order to have less biased results.
In summary, thinking about quality issues will help us to have higher quality studies, which could help policy makers to make better decisions and clinicians to improve the health of their patients.
Demián Glujovsky, M.D., M.Sc.
Center for Studies in Gynecology and Reproduction (CEGYR) and Institute for Clinical Effectiveness and Health Policy (IECS)
Buenos Aires, Argentina
Agustín Ciapponi, M.D., M.Sc.
Institute for Clinical Effectiveness and Health Policy (IECS)
Buenos Aires, Argentina
1. World Health Organization. World Health Organization international clinical trials registry platform. New standards for registration of human medical research. Geneva, Switzerland: World Health Organization; 2006. Available at http://www.who.int/mediacentre/news/releases/2006/pr25/en/. Accessed January 10, 2012.
2. Booth A, Clarke M, Ghersi D, Moher D, Petticrew M, Stewart L. An international registry of systematic-review protocols. Lancet 2011;377:108-9.
3. Van der Wees P, Qaseem A, Kaila M, Ollenschlaeger G, Rosenfeld R, Board of Trustees of the Guidelines International N. Prospective systematic review registration: perspective from the Guidelines International Network (G-I-N). Systematic reviews 2012;1:3.
Published online in Fertility and Sterility doi:10.1016/j.fertnstert.2013.01.119