When Scientific Methods Fail: New Criticisms Over the Wax et al Homebirth vs. Hospital Birth Study

Just when one might think the controversy over the Wax et al planned homebirth vs. planned hospital birth study might be dying down, it is instead heating up again.

The American Journal of Gynecology—in which the study was originally published in September, 2010—released its April 2011 issue last Friday, full of Letters to the Editor criticizing the study.  Written by social science, epidemiology, bioethics and public health doctoral university researchers and midwives, it almost seemed as if the journal was preparing to retract the Wax article by these inclusions.  But really, they were providing the platform for the study authors to respond to the critical letters, as suggested in an excerpt from one of those responses:

“Although our findings may be unpopular in certain quarters, they result from appropriate rigorous scientific methods that have undergone appropriate peer review.”

(More on the “scientific methods” later.)

In the concluding Editor’s comment, we are told an (anonymous) independent panel of maternal fetal medicine experts convened to review the data in the Wax study—a move to theoretically decide whether or not the journal should retract the article.  The panel’s recommendations, following this analysis of the Wax et al data, were that AJOG should publish online, a full summary of the graphs for each outcome included in the study and that no retraction of the article is warranted.  (They did admit to finding ‘minor’ discrepancies in the data—but none that seemed to warrant rescinding the article altogether.)

I will say, I have to give AJOG a little credit here:  they didn’t have to print those letters.  I can only imagine how many letters in a month a large journal like that receives.  The managing and chief editors certainly could have kept all those letters to themselves, never to print a single one.  Perhaps their PR department convinced them that doing so would have created a larger fire storm than the one (still) brewing.

Coming on the heels of articles in Nature and the Lancet, which seriously called into question the conclusions of the Wax study, there has certainly been a lot of pressure on AJOG to address its decision to publish the article at all.

Medscape is also on board the debate—having already re-published the Nature article and now making available a rebuttal article from a group of researchers—including the authors of the British Columbia and Netherlands homebirth studies the Wax article incorrectly attributed and irresponsibly excluded, respectively.  This latest article, entitled Planned Home vs. Hospital Birth: A Meta-Analysis Gone Wrong, by Carl Michal, PhD, et al, was also published last Friday.

An excerpt from the Michal paper encapsulates the problem with the Wax meta-analysis:

The statistical analysis upon which this conclusion was based was deeply flawed, containing many numerical errors, improper inclusion and exclusion of studies, mischaracterization of cited works, and logical impossibilities. In addition, the software tool used for nearly two thirds of the meta-analysis calculations contains serious errors that can dramatically underestimate confidence intervals (CIs), and this resulted in at least 1 spuriously statistically significant result. Despite the publication of statements and commentaries querying the reliability of the findings, this faulty study now forms the evidentiary basis for an American College of Obstetricians and Gynecologists Committee Opinion, meaning that its results are being presented to expectant parents as the state-of-the-art in home birth safety research.

Critical analysis of the Wax study by Michal et al includes the following:
°data included in the study suggests a higher neonatal death rate (for both home and hospital births) compared to perinatal death rates.  This, of course, is not possible as neonatal deaths ought to be included in the perinatal death numbers—therefore the data here are paradoxical in nature.

°multiple numerical errors including inconsistencies in data provided in both original and supplemental source papers; incompatibilities between data from cited sources and that which is represented in the Wax study

°inconsistencies in the authors’ definitions of perinatal and neonatal mortality

°miscalculations of some odds ratios (OR) and confidence intervals (CI)—sometimes to the tune of drastic underreporting of the CI—resulting in inaccurate statistically significant positive or negative results

°inappropriate data inclusion criteria (such as that for perineal tears, in which the Wax study only included data on first and second degree tears, rather than all perineal lacerations)

°the meta-analysis spreadsheet used to calculate 13 out of the 21 outcomes contained a computational error—making all data computed with that spreadsheet incorrect

°inclusion of reference works that, themselves, have been highly criticized for statistical inaccuracies (such as the Pang et al study that included unplanned home births when it had set out to only assess planned homebirths)

°misappropriated causation:

…the discussion of causes of neonatal mortality focuses on findings from studies that were not included in the meta-analysis, including studies that mix high-risk with low-risk cases. Of the studies that are included in the meta-analysis, none associates rates of intervention with rates of neonatal mortality.

AJOG, in an attempt to rule on the debate of whether or not the data presented in the Wax study are even valid, may have just shot itself in the foot.  Since when did ‘we published an inaccurate study, and we stand by those inaccuracies’ become acceptable?


(see appropriate references attached to above-linked articles)

Posted by:  Kimmelin Hull, PA, LCCE

Home Birth, Metaanalyses, Research , , , , , , ,

  1. avatar
    April 5th, 2011 at 11:32 | #1

    I got to hear Michele Odent speaking about this study last week and was really fascinated to hear his take. Rather than criticizing the study methodology, he said that its results as well as a couple of others of the last 6 months (Netherlands and China) indicate that birth is becoming more difficult in all settings. He suggests that we have a gross understanding of the needs of a laboring woman across the board, even in natural childbirth circles. I’ll be writing more about this in the coming weeks because it is possible that we are largely turning a blind eye to a lot of widely accepted and in fact celebrated and endorsed practices even in the most natural and low-intervention approaches that may not actually be physiologically sound.

    That being said, I’d be delighted to see this study retracted or taken down as inflammatory or unsound – but if there IS a kernel of truth in it, I think we’re wise to look directly and honestly and the implications.

  2. avatar
  3. April 5th, 2011 at 16:10 | #3

    Thank you for sharing the link, Tatiana.

  4. April 6th, 2011 at 22:31 | #4

    I don’t think Wax will get retracted. Criticism of controversial studies is part of the scientific publishing process, which is why AJOG published those letters. They get hundreds of submissions a month, and in this case they didn’t rigorously evaluate the numbers that were submitted. Peer review generally doesn’t get to that standard. I peer review articles all the time, and typically I don’t take it upon myself to redo all the author’s stats. In a study like this, few peer reviewers would have had the expertise to do so.

    AJOG doesn’t have some internal agenda, other than perhaps to sell ads. It is a peer reviewed journal, and a fairly low impact one at that. AJOG isn’t nearly a strong a journal as the Green, and way below something big like Lancet or NEJM. They don’t have some political agenda. They just want to publish the things that come across their desk that appear important and reasonably well written.

  5. avatar
    Jo Anne Davis
    April 7th, 2011 at 12:09 | #5

    Indeed, the controversy thickens! I agree with Odent’s concerns, the challenges to the rigor of the Wax, et.al., study, and the need for enhanced oversight of research publication.
    The Elsevier anonymous review panel cannot, on the face of it, be viewed as completely objective. Conscious or not, ideological bias of educational and practice philosophy continue to influence analysis. This is true in all research endeavors, though most will not admit to it.

    And here is the bottom line: With all due respect, any who purport to address the issue of planned, out of hospital births that are attended by a skilled and licensed attendant, must be viewed as failing to possess the necessary authority and credibility, unless she or he possesses extensive out of hospital birth experience.

  6. April 8th, 2011 at 16:18 | #6

    @Nicholas Fogelson, MD
    I have had experience with the Am J of OBS and Gyn in the past, and have found them open to correct controversial information in the service of excellence. For example when I submitted my RCT of episiotomy, showing that routine episiotomy caused the very problems that it was suposed to prevent, their reviewers trashed the work as lacking in importance, and two of the reviewers inserted misogynistic comments. The journal declined to publish. I called the editor and brought to his attention the inappropriate and biased comments of the reviewers. He sent out the paper to three new reviewers. The journal then published the paper and two followup papers. BUT in the Wax case, the paper appeared as the “EDITORS CHOICE.” That says to me that the editor read the paper, liked what it said and really wanted the audience to read it. Why did he like it? If you think that he wanted controversy and that is why, then your believe in the incompetence theory of history. If you think that it was editors choice because it fit with the political agenda of the journal and ACOG, then you believe in the devil theory of history. Take your choice.

    Michael C. Klein, MD,CCFP,FAAP (Neonatal-Perinatal), FCFP,ABFP,FCPS
    Emeritus Professor Family Practice & Pediatrics
    Director Clinician Scholars Program
    Department Family Practice, University British Columbia
    Sr. Scientist Emeritus
    Ctr Developmental Neurosciences & Child Hlth
    Child and Family Research Institute
    Listmaster: Maternity Care Discussion Group (MCDG)

  7. April 14th, 2011 at 18:37 | #7

    @Michael Klein
    The Grey Journal (AJOG) and ACOG? AJOG is not connected to ACOG in any way. ACOG publishes the Green Journal, not the Grey.

    I suspect it was more incompetence than anything else. The same could be said for publication of the WHI or the TBT. It seems to me that if any study reaches a large enough magnitude, it pretty much will get published on that basis alone. Journals complete to publish studies that are deemed to be “significant”, even if those significant studies are fatally flawed.

  8. avatar
    February 6th, 2012 at 14:02 | #8

    “data included in the study suggests a higher neonatal death rate (for both home and hospital births) compared to perinatal death rates. This, of course, is not possible as neonatal deaths ought to be included in the perinatal death numbers—therefore the data here are paradoxical in nature.”

    Isn’t the definition of “neonatal death” a death occurring in the first month whereas “perinatal death” is death in the first week? If so, the neonatal numbers would be higher than or equal to the perinatal numbers and this isn’t paradoxical at all.

  1. February 23rd, 2012 at 09:01 | #1
  2. October 18th, 2012 at 15:22 | #2