Meta-analysis: the wrong tool (wielded improperly)

A lot has been said about the new meta-analysis of home birth. (Here is an excellent summary from Jennifer Block.) Canadian physician Michael Klein has been widely quoted as saying that the meta-analysis, a potentially valuable statistical tool, was performed poorly because the researchers included studies using discredited methodology, as well as studies that are decades old. “Garbage in, garbage out.” I totally agree with this assessment. I also take issue with the fact that the researchers did not display the standard “forest plot” that customarily accompanies a meta-analysis to illustrate how the relative magnitude of observed differences in the individual studies and the pooled analysis. And I’m perplexed by the use of a fixed-effects model for the analysis of neonatal death.

But I want to take a step back and ask a larger question - is meta-analysis even appropriate for the study of home birth?

Meta-analysis is a statistical process that pools data from multiple studies. It is intended to achieve two related goals:

  • have adequate statistical power to detect differences in rare but clinically important outcomes (such as perinatal mortality among babies of healthy women)
  • establish a definitive answer to an important clinical question, so that policies and practices can adapt to conform to the new “truth” and other researchers don’t have to study the issue anymore.

Let’s look at these two issues separately in the context of the Wax meta-analysis.

Statistical Power

Lack of statistical power could not possibly be the rationale for conducting a meta-analysis on the safety of home birth. That’s because there already is a study large enough to detect differences in intrapartum and neonatal death. In fact, it contributed 94% of the data on planned home birth in the meta-analysis (321,307 of 342,056 planned home births). That study found virtually identical rates of neonatal death in both the planned home and planned hospital births*, with relatively narrow confidence intervals. Neonatal deaths on day 0-7 occurred in 3.4 per 10,000 of each group and when combined with intrapartum mortality and adjusted for confounding factors, the relative risk was 1.00 (95% CI 0.78 to 1.27). That means that there was a 95% likelihood that planned home birth results in somewhere between a 22% reduction and a 27% increase in intrapartum or neonatal mortality.)

By adding a bunch of smaller, older, and flawed studies, excluding the intrapartum deaths (which may be affected by intrapartum events and therefore are potentially modifiable by the birth setting) and adding deaths that occurred between 8-28 days (which are less likely to be related to intrapartum events and therefore are less modifiable by birth setting), we suddenly have nearly three times the neonatal mortality rate with planned home birth and a confidence interval you could drive a truck through?  (a 95% chance that home birth increases the risk of neonatal death by somewhere between 32% and 625%)  Hmmm…

Definitive “truth”

The other reason to undertake meta-analysis is to definitively settle a clinical question. Meta-analysis, after all, holds a privileged place atop the evidence pyramid, where it is considered the “best evidence.”  But is a deeply flawed meta-analysis really better than an adequately powered, methodologically sound study? The answer, of course, is no. All the meta-analysis does in such cases is separate the reader from the primary source of the data so that they can’t assess it for themselves, while putting the evidence-based stamp of approval on whatever statistics the meta-analysis software spits out. But people with a political motivation to authoritatively declare a certain definitive truth may realize that most people don’t bother to check to see if a meta-analysis is done appropriately or critically assess the quality of the included studies. They just go, “Oh look, there’s a meta-analysis of home birth and it said it’s 3 times riskier than hospital birth. That settles that! It’s a meta-analysis, after all!”

So if not a meta-analysis, then what?

OK, so if meta-analysis was not the right tool, what is?  And can we stop studying the safety of home birth now that we have that large study that contributed 94% of the home birth data to the meta-analysis?

The way I see it, the large study that showed equivalent perinatal outcomes between home and hospital birth tells us definitively that home birth can be safe. But it doesn’t tell us that home birth is intrinsically safe. We need to continue to study home birth using all of the tools in the research toolbox, qualitative and quantitative, to determine under what circumstances home birth is safe and how to optimize care and outcomes in all birth settings. And we need to stop pushing home birth underground in the United States where it remains a fringe alternative, poorly integrated with the maternity care system, with no standard safety net in place for women who begin labor with the intention to birth at home but turn out to need hospitalization in order to birth safely. Shame on the American Journal of Obstetrics and Gynecology for making this task even more difficult than it already was, by publishing and publicizing a junk meta-analysis.

*edited 7/12/2010 to correct a (serious) error. Sentence previously read “virtually identical rates of neonatal death in both the planned and unplanned home births.”

Uncategorized , , , ,

  1. avatar
    | #1

    @Amy Tuteur, MD
    You are changing the goalposts here, the original comment by “b” said nothing about the EU.
    EU does not = “all of Europe”
    Facts are facts, geography is geography, and hyperbole is hyperbole.
    The Netherlands does not have the worst perinatal mortality in all of Europe. As a matter of fact, it does not have, as you claim “one of the worst perinatal mortality rates in Europe”. There are 15 countries in Europe that are worse. If you can point me to more recent data than the 2006 WHO report, please do.
    Any reasonable person reading that original statement would assume that “Europe” = “Europe” not “Europe” = EU.
    The fact perinatal mortality in the Netherlands is somewhat higher than most of its nearest neighbors does not suddenly change the accepted definition of what constitutes Europe.

  2. | #2

    The Dutch believe that their high perinatal mortality rate is a serious problem and are actively investigating to determine the cuse. The government has commissioned researchers at the Erasmus Medical Centre in Rotterdam to oversee the investigation. From the Erasmus MC website:

    “The Netherlands has a relatively poor position in Europe when it comes to health at the time of birth, in other words, perinatal health. Approximately 10 out of every 1000 children die around the time of birth. In similar other countries this mortality rate can be as much as 30% lower. Of the perinatal deaths in the Netherlands, 70% are stillbirths when counted from the 22nd week of pregnancy. Thirty percent of the perinatal deaths take place in the first week after birth. In Flanders, that is socio-democratically and economically comparable to the Netherlands, the perinatal death rate has been two-thirds of that in the Netherlands for at least 10 years. This means that instead of 1700 cases of perinatal death that occur per year among the 175,000 newborns in the Netherlands, only 1150 cases should occur; an unprecedented large difference. Moreover, within the Netherlands, and particularly in the larger cities such as Rotterdam and The Hague, there are distinct differences between groups of pregnant women.

    The ZonMw has commissioned Erasmus MC to carry out the Descriptive study Pregnancy and Childbirth. The aim of the study is to determine knowledge questions and research opportunities to improve the perinatal care in the Netherlands. Aspects studied include patient-related risk factors such as diseases already present, lifestyle and social factors on the one hand and the role of the midwife practices including use of care, risk selection, and quality of care in the Netherlands on the other. The preliminary conclusion is that the unfavorable European position is probably mainly caused by factors in the care system while the differences within the Netherlands and the larger cities are linked to large risk differences between groups on the basis of ethnicity, social deprivation and the neighborhood in which people live. A research agenda has been formulated based on this.”

    In other words, the government investigation found that one of the main reasons for the high perinatal death rate is the midwife care system including use of care, risk selection, and quality of care.

  3. avatar
    | #3

    Amy Tuteur, here is an example of how you twist the information. It says “factors in the care system”, which you have specifically pinned down to “midwife care system” for some reason. Are there no other factors in their care system besides the midwives?

    Midwifery care is best for low-risk mothers and babies. If mothers are not being screened properly and shifted to OB and hospital care in the event of increased risk during the pregnancy and/or birth, then that is a problem. It is possible for the Netherlands to implement more thorough screening and collaboration between midwives and OBs and yet still have a higher home birth rate and lower perinatal mortality rate than other countries. Changes need to be made, and good research needs to be done.

  4. avatar
    | #4

    @Amy Tuteur, MD
    So let me start out with another clarification of less-than-accurate use of language:
    “There is simply no question that the country with the highest homebirth rate in the world has the one of the worst perinatal mortality rates in Europe.”

    Actually, I would bet that the country with the highest homebirth rate in the *world* is not even in Europe. I’m sure that what you meant to say is:
    “There is simply no question that the country with the highest homebirth rate in the ***developed*** world has the one of the worst perinatal mortality rates in ***Western*** Europe.”

    And leaving off the qualifiers does actually affect the accuracy of the statement. Maybe it’s just a loose shorthand, but you should be aware that some readers might interpret this casual dismissal of the rest of the world as reflecting a distorted worldview. Whatever.

    On to the substance: So, it’s true that the country with the highest homebirth rate in the developed world has the one of the worst perinatal mortality rates in Western Europe. How can we determine if those two observations are causally related? Well, we can combine the numbers provided by the Erasmus MC with the results reported in the 2009 deJong paper, which showed a total of 207 intrapartum and neonatal deaths in the planned home birth group, over a period of 7 years. This is fewer than 30 deaths per year on average. So even if you made the assumption that every one of these deaths could have been prevented by more optimal care (which is quite implausible), this would potentially lower the overall perinatal death rate from 9.7 per thousand to 9.5 per thousand births. More realistically, it’s probably no more than a rounding error.
    Even looking at the controls in the deJong study (planned hospital births in women who met selection criteria for home birth) would only account for another 16 – 17 deaths per year. So perhaps another 0.1 per thousand reduction.
    So what we can determine is that the excess perinatal deaths in the Netherlands are almost all occurring in the 58% of women who would not have been considered as home birth candidates.
    Now it is certainly *possible* that the primary care midwives are responsible for some of these deaths – if they are not making timely referrals, or not catching potential complications early enough. But there are many other “factors in the care system” besides midwives, which may contribute to the higher than optimal death rates associated with medium-to-high risk hospital births in the Netherlands.

    And the largest factor may be the Dutch attitudes regarding aggressive care for the most frail newborns.
    “The very preterm births (22.0-25.6 weeks of gestation) provided 29% of all perinatal mortality with a mortality risk of 935 per 1000 births.”

    Since the well-screened low risk part of the birthing population (42%) accounts for less than 3% of the perinatal deaths in the Netherlands, and this population has equally low rates regardless of planned birth location, it is simply not credible to argue that the availability and support for home birth is a cause for any significant part of the national perinatal mortality rate.

  5. | #5

    The Jennifer Block quote is not working. Any chance you might have current link?

  6. | #6

    @Penny Penny, I wish I did, but you might need to connect with Jennifer Block to find out, Sorry. Sharon

Comment pages
  1. | #1
  2. | #2
  3. | #3
  4. | #4
  5. | #5