To the Editor:
We read with great interest the article by Goldman et al. (1). We would like to make some comments regarding the methodological and statistical issues raised by this work.
First, in our opinion, the outcome measure is not accurate enough to evaluate skill acquisition in oocyte retrieval (OR). A successful OR should be defined by the ability for the trainee to perform the procedure without intervention of the tutor as well as to retrieve the expected number of oocytes. However in this study, cases where more than one provider attempted OR on one ovary were excluded from analysis. What about the cases where the trainees failed to complete OR alone? Were they not taken into account in the learning curve?
As the authors admit, another methodological limitation was that the senior operator chose which ovary was retrieved by the trainee. For that reason, the trainees might have performed only the good cases and their scores might be over evaluated. Randomization of the side would have been more appropriate.
Second, the statistical methods used are also questionable. In the first part of the study, the authors compared the procedures, per group of 10, pooled among trainees. However, it is well acknowledged that trainees learn at a different pace (2, 4, 5) and pooling their results is likely to smooth relevant differences. Then, choosing an arbitrary cut-off is likely to average the learning curve over a period of time. Therefore, it is not surprising that the authors did not find any statistical significant difference between trainees and seniors, nor between trainees over the different periods. In the second part of their study, the authors attempted to characterize individual trends in the learning curve with inappropriate statistical tools. The use of linear regression models to fit an unlikely bell-shaped learning curve is limited and the comparison of one trainee’s performance over time and of one trainee to another suffers great limitation given the potentially wide confidence interval.
Overall, the authors observed no statistical difference in the learning curve between fellows. These results contrast with published data showing that in the field of surgery (3, 4), ultrasound diagnosis (5), and reproductive medicine (1), the length of the LC varies widely between individuals. We believe that measuring the number of procedures required by the average trainee to reach proficiency suffers the same old caveats: once you find that threshold, half the trainees will still be considered competent to perform a procedure when they are not.
The right approach in quality control would seem to us to implement an individual prospective monitoring of the LC. Moreover, after a trainee is considered proficient, monitoring should continue to ensure that performance is maintained. The cumulative summation test for learning curve (LC-CUSUM test) was specifically developed for that purpose (6) and has been used with success in different settings.
Lionel Dessolle, M.D., M.Sc.
David Biau, M.D., Ph.D.
1. Goldman KN, Moon KS, Yauger BJ, Payson MD, Segars JH, Stegmann BJ. Proficiency in oocyte retrieval: how many procedures are necessary for training? Fertil Steril, in Press.
2. Dessolle L, Fréour T, Barrière P, Ravel C, Daraï E, Jean M, Biau DJ. How soon can I be proficient in embryo transfer? Lessons from the cumulative summation test for learning curve. Hum Reprod 2010; 25(2):380-386.
3. Biau DJ, Williams SM, Schlup MM, Nizard RS, Porcher R. quantitative and individualized assessment of the learning curve using LC-CUSUM. Br J Surg 2008; 95(7): 925-929.
4. Papanna R, Biau DJ, Mann LK, Johnson A, Moise KJ Jr. Use of the Learning Curve-Cumulative Summation test for quantitative and individualized assessment of competency of a surgical procedure in obstetrics and gynecology: fetoscopic laser ablation as a model. Am J Obstet Gynecol 2011; 204(3): e1-9.
5. Bazot M, Daraï E, Biau DJ, Ballester M, Dessolle L. Learning curve of transvaginal ultrasound for the diagnosis of endometriomas assessed by the cumulative summation test for learning curve. Fertil Steril 2011:95(1):301-303.
6. Biau DJ, Porcher R. A method for evaluating a process from an out of control to an in control state: application to the learning curve. Stat Med 2010; 29(18): 1900-1909.
Published online in Fertility and Sterility doi:10.1016/j.fertnstert.2011.06.018
The Authors Respond:
We thank the respondents for their interest in our manuscript. We have reviewed the LC-CUSUM model designed by Biau (1,2) with great interest, and while the respondents make some interesting points, the unique aspects of oocyte retrieval (OR) make it virtually impossible to apply Biau’s model to our dataset.
Primarily, the outcome of interest for LC-CUSUM has to be based in the success/failure of retrieving an oocyte from an individual follicle identified on an ultrasound two days prior to surgery. This is simply not possible. In addition, failure to retrieve an oocyte is possible for many reasons that are unrelated to operator technique, and this model would unfairly penalize the trainee when that occurred.
Furthermore, retrieval of more oocytes than expected is possible as smaller follicles can unexpectedly produce mature oocytes. In this circumstance the trainee would be unjustly rewarded. We compensated for such problems with outcome misclassification by smoothing/pooling results over 10 retrievals. Our approach minimized the effect of both good and bad results, thus lessening the impact of errors in the outcome assignments.
The respondents also suggested it would have been more appropriate to randomize the side of the retrieval. While this would have been correct if we were performing a study, this was not a randomized clinical trial but a retrospective review of existing data from a training program. In a similar vein, an important part of a training program is the development of good clinical judgment. Labeling a procedure as a “failure” because a fellow used good clinical judgment and requested assistance instead of placing a patient at increased risk would be incorrect. Therefore, our decision to discard the very few procedures in which a fellow did not complete a procedure based on clinical decisions is not only logical, but was in the interest of good patient care.
The purpose of this study was not solely to determine competence on an individual level; instead, our goal was to provide information for training programs. Specifically, our objective was to determine the minimal number of procedures needed for the majority of trainees to become competent. While LC-CUSUM clearly has advantages in situations where competence is being evaluated on an individual level (3-5), it does not provide global program information. We agree that once a skill is acquired, continued monitoring is needed to ensure the maintenance of the skill, and Dr. Biau’s model is certainly an excellent tool when prospectively evaluating surgical skills or ultrasound interpretation. However, its usefulness in datasets such as ours, where outcomes are less clearly defined, is limited.
Barbara Jean Stegmann, M.D., M.P.H.
Department of Obstetrics and Gynecology
Division of Reproductive Endocrinology and Infertility
University of Iowa Hospitals and Clinics
Iowa City, Iowa
1. Biau DJ, Porcher R. A method for monitoring a process from an out of control to an in control state: Application to the learning curve. Stat Med 2010;29:1900-9.
2. Biau DJ, Williams SM, Schlup MM, Nizard RS, Porcher R. Quantitative and individualized assessment of the learning curve using LC-CUSUM. Br J Surg 2008;95:925-9.
3. Papanna R, Biau DJ, Mann LK, Johnson A, Moise KJ, Jr. Use of the Learning Curve-Cumulative Summation test for quantitative and individualized assessment of competency of a surgical procedure in obstetrics and gynecology: fetoscopic laser ablation as a model. Am J Obstet Gynecol 2011;204:218 e1-9.
4. Dessolle L, Freour T, Barriere P, et al. How soon can I be proficient in embryo transfer? Lessons from the cumulative summation test for learning curve (LC-CUSUM). Hum Reprod 2010;25:380-6.
5. Bazot M, Darai E, Biau DJ, Ballester M, Dessolle L. Learning curve of transvaginal ultrasound for the diagnosis of endometriomas assessed by the cumulative summation test (LC-CUSUM). Fertil Steril 2011;95:301-3.
Published online in Fertility and Sterility doi:10.1016/j.fertnstert.2011.06.019