24h-payday

Archive

Posts Tagged ‘meta-analysis’

The Wax Home Birth Meta-Analysis: An Outsider’s Critique

October 23rd, 2012 by avatar

Today’s post is a fascinating interview that took place between Rebecca Dekker, PhD, RN, APRN of Evidence Based Birth and Kyoung Suk Lee, PhD, MPH, RN, APRN. Rebecca asked Dr. Lee to provide a review of the Wax Home Birth Meta-Analysis, as an “unbiased outsider”, but highly skilled researcher.  Dr. Lee’s comments and critique are fascinating and provided me with many further thoughts.  Please enjoy Rebecca’s interview and share your comments. – SM

__________________

http://www.flickr.com/photos/eyeliam/7353095052/

Shortly after starting my website, www.evidencebasedbirth.com, I had several people ask me if I could write an article about the research evidence on home birth. However, I was hesitant to do so for several reasons. Mainly, I was worried that I could not look at the evidence in an objective manner. My husband and I had recently chosen a home birth for our second child. I was worried that it would be difficult to objectively examine the research evidence on home birth, given my personal experience. The blogosphere is full of people who are strongly pro-home birth or anti-home birth, and their evaluations of the evidence are usually written through the lens of their own biases. I didn’t want to add to the plethora of biased articles already out there.

Then I had a sudden burst of inspiration. What if I asked one of my colleagues—who has no biases about childbirth—to review the home birth literature for me? In particular, I wanted to find someone who could review the Wax home birth meta-analysis (Wax, Lucas et al. 2010) and give me a fair assessment of its scientific value.

I chose the Wax meta-analysis for this review because in 2011, the American Congress of Obstetricians and Gynecologists emphasized the results of the Wax study in its official statement on home birth. Their statement said: “Women inquiring about planned home birth should be informed of its risks and benefits based on recent evidence. Specifically, they should be informed that although the absolute risk may be low, planned home birth is associated with a twofold to threefold increased risk of neonatal death when compared with planned hospital birth.”(ACOG, 2011)

Dr. Kyoung Suk Lee, PhD, MPH, RN, APRN

It did not take me long to figure out who I would ask to review the Wax study. Dr. Kyoung Suk Lee is considered by her colleagues to be a rising star in the field of cardiovascular research. She recently graduated with a PhD in Nursing, and she just accepted a job at a research university. People who work with Dr. Lee say that she is extremely intelligent, hard-working, and a future leader in her field. Dr. Lee’s expertise has been recognized with research awards from the Heart Failure Society of America, the Society for Heart-Brain Medicine, and the Cleveland Clinic Heart-Brain Institute, among others. She has published her work in nursing and cardiology journals. Furthermore, I knew that Dr. Lee did not have any biases about childbirth, home birth, or hospital birth. I asked Dr. Lee if she would be willing to review the Wax meta-analysis for me, and she kindly agreed.

What follows is my interview of her about the study and its results (RD in bold, KSL unbolded).

Do you have any biases or conflicts of interest related to home or hospital birth?

I do not have any biases related to home or hospital birth.

Could you summarize the methods and results of the Wax study?

The purpose of this meta-analysis was to compare maternal and neonatal outcomes between planned home-and hospital-births.

Using an electronic database search and bibliography search, the authors retrieved 237 articles and included 12 articles in their meta-analyses. Of 12 articles included, 3 were conducted after 2000 while 9 were conducted before 2000. Of 12 articles, 2 were conducted in the US (one was a retrospective design) while 10 were conducted outside US.

Women in the planned home birth group had better maternal outcomes than women in the planned hospital group. They had fewer interventions such as epidurals and episiotomies, and lower morbidity (infection, 3rd or 4th degree lacerations, hemorrhages, and retained placenta). There were no differences in cord prolapse between the two groups.

For neonatal outcomes, babies born to women in the planned home birth group were less likely to experience prematurity and low birth weight. However, babies born to women in the planned home birth group were more likely to experience neonatal death compared to women in hospital birth.

What is the difference between neonatal and perinatal mortality? What does this have to do with the results?

Based on the definitions given by the authors, neonatal mortality was defined as “death of live born child within 28 days of birth.” This is a subset of an overall outcome– perinatal mortality, which was defined as “stillbirth (of at least 20 weeks or 500g) or death of live born child within 28 days of birth.”

According to the authors, there were no differences in perinatal death (the overall outcome) between planned home birth and hospital birth groups. However, homebirth was associated with 2 times higher risk for neonatal death (the subset of deaths occurring 28 days after birth) in all infants and 3 times higher risk for neonatal death in infants who did not have any congenital birth defects.

Interestingly, if you look at page 243.e3, the authors did a sensitivity analysis. In this analysis, they excluded the studies that had home births that were not attended by certified midwives or certified nurse midwives. In this analysis, they found that there were no differences in neonatal deaths between the home birth and hospital birth groups. This means that in the studies in which midwives with certification of some kind attended home births, the outcomes were the same except there was no increase in the neonatal death rate. In my opinion, we have to pay attention to results of sensitivity analyses because this allows us to see the results based on studies which were definitely known to be eligible or clearly described their methods and outcomes.

What is your opinion on the scientific rigor of this meta-analysis?

One thing that was strange to me is the odds ratios (ORs) in the tables. For example, in table 2, under morbidity, the percentages of infection between home births and hospital births were 0.7 vs. 2.6 (its OR was 0.27) while percentages of perineal laceration were 42.7 vs. 37.1 (its OR was 0.66). To a researcher, these numbers don’t make sense.

Many of the studies included were older (half of the studies were conducted more than 20 years ago) so results may not reflect the current practice at home births or hospital births.

The authors did not provide detailed information on how they evaluated the quality of studies included, although they cited a paper describing the method of study evaluation. This makes it difficult if not impossible to determine whether the studies they included were of good or poor quality.

The authors mentioned that women with high risks would prefer hospital births so that it would expect that home births have better outcomes than hospital births in some maternal and neonatal outcomes. If this was a concern, I wonder why the authors didn’t just focus on only the studies that used matching methods, in order to minimize confounding factors.

What is the difference between relative risk and absolute risk, and how does that apply to women who want to have a home birth?

Absolute risk is the probability of something occurring. They may be expressed as percentages or ratios. For example, neonatal mortality rate in the United States is 2.01 per 1,000 live births. This is .201 percent (2.01/1000 = .201/100).

http://www.flickr.com/photos/mikeporcenaluk/3789756395/

Relative risk is a comparison between different risk levels, such as the neonatal mortality rate of home birth compared to the neonatal mortality rate of hospital birth. The researchers found that there was a higher relative risk in neonatal mortality at home births compared to hospital births, but the overall absolute risk for both was small.

How can women know whether the Wax study results would be applicable to their own individual situation?

Meta analysis is one way to generalize findings from different studies. However, women and clinicians should interpret these results cautiously because the studies included were very different from one another and some of the studies included may not have been of good quality. Also, it would be important to note that the overall neonatal death rate that they report reflects home births that were attended by midwives as well as those that may not have had any kind of certified midwife present.

Because this study seems to have some flaws, the conclusion is tentative. I do not know if this article has any implications for pregnant women.

What do you think is the value of asking someone with no conflicts of interest to evaluate controversial research? Does Dr. Lee’s even-handed critique make you view the results of this study any differently? How do you feel about Dr Lee’s conclusion that the study’s results are tentative, and that the Wax study might not have any implications for pregnant women? Please share your thoughts and comments with other readers.

References

(2011). “ACOG Committee Opinion No. 476: Planned home birth.” Obstetrics and gynecology 117(2 Pt 1): 425-428.

Wax, J. R., F. L. Lucas, et al. (2010). “Maternal and newborn outcomes in planned home birth vs planned hospital births: a metaanalysis.” Am J Obstet Gynecol 203(3): 243 e241-248.

About Rebecca Dekker

Rebecca Dekker, PhD, RN, APRN, is an Assistant Professor of Nursing at a research-intensive university and author of www.evidencebasedbirth.com. Rebecca’s vision is to promote evidence-based birth practices among consumers and clinicians worldwide. She publishes summaries of birth evidence using a Question and Answer style.

Babies, Childbirth Education, Evidence Based Medicine, Guest Posts, Home Birth, Metaanalyses, Midwifery, New Research, NICU, Research , , , , , , , , , , , ,

Becoming a Critical Reader: Questions to ask about systemic reviews and meta-analyses

November 14th, 2010 by avatar

Systemic reviews are generally considered to be at the top of the evidence pyramid, providing one of the best sources of information. But just like any other type of research, a systemic review is only as good as the work and data that goes into it. A systemic review carefully looks at all of the evidence using a rigorous, predefined system of methods, and draws conclusions based on the information gathered. That rigorous, predefined system of methods is critical to a good systemic review. The researchers go through several steps:

  • Select a specific, well-defined question
  • Lay out their criteria for searching and selecting the evidence
  • Conduct a very thorough search of all the available literature
  • Evaluate the studies, rejecting the studies that are of poor quality
  • Review the studies that make the cut
  • Make a recommendation for practice

A systemic review differs from a general review of the literature in the methodology used. A systemic review starts by formulating specific criteria that will be used to judge which studies will be included and which will be excluded. The criteria are set before any of the studies are reviewed; ideally this will prevent bias and make for a stronger, more valid result.

Some systemic reviews include a meta-analysis, where statistical techniques are used to combine the results of the included studies and use the larger sample size to draw a stronger conclusion. But don’t assume that all meta-analyses use the systemic review process! It’s entirely possible to conduct a meta-analysis of a group of studies chosen in an incomplete or biased manner. The questions below can help you identify which meta-analyses use the systemic review process.

When reading a systemic review or meta-analysis, here are some questions to consider:

1. How well is the question defined? There should be a clear statement of what the review would like to show. Then double check the results to make sure that question actually got answered.

2. Is it the right question? If two things are being compared, is the comparison appropriate? Do all the studies included use the same comparison/control?

3. Do the authors describe their search? Was it thorough? Authors should discuss how they went about searching for the articles they evaluated. A thorough researcher will look at multiple databases, use variations of the key words, and include studies published in other languages.  Limiting to studies published in English is convenient, but you may miss valuable research. Unpublished studies are also sometimes included, as it can be difficult to get a study published if your results showed no dramatic differences. This can help avoid publication bias, but unpublished studies should still be thoroughly checked for quality. The search and selection methods should be so clearly outlined that someone could duplicate them.

The Cochrane Collaboration is best known for conducting systemic reviews. In the Cochrane organization, reviewers publish their protocols before conducting the review. (If you are a Lamaze member, you can access these protocols on the Cochrane site by logging in through the Lamaze Member Center.) This is not generally done elsewhere, but in the published paper, the reviewers should explain their protocol. Any potential conflicts of interest should also be addressed.

4. Did the authors evaluate the quality of the studies reviewed? Not all studies are of equal quality, though sometimes studies with quality issues can still provide useful information. One example of this is the Cochrane Review on skin-to-skin, which included some studies that did not have completely random groupings if it appeared that the groups were otherwise equal.  For this reason, many systemic reviews will rate the quality, size and applicability of the studies as they evaluate them, and assign them a weight so that the most appropriate studies are more heavily represented in the results.

5. Are there any biases in the inclusion/exclusion criteria? Read through them very carefully and evaluate this aspect. Easier when you’re already familiar with the studies out there, or if the excluded studies are listed for your viewing. Overly restrictive criteria lead to smaller sample sizes and less reliable results.

6. Were the outcome measures clearly defined? What are the benefits or risks the researchers were looking at? Are outcomes lumped into groupings of debatable usefulness? Do they matter? As Amy Romano recently pointed out sometimes the outcomes have little or no real life importance.

7. Are there biases in how subgroups are analyzed? In one interesting study researchers created random data with the roll of the dice. The studies used dice that were identical other than color, and each time a 6 was rolled, it was counted as a patient death. The researchers then manipulated subgroup analysis to show that red dice had significantly higher death rates than other colors!

Don’t assume that the general inclusion/exclusion criteria were applied to each subgroup. As we learned with the Wax meta-analysis on home birth, selective subgroup analysis does happen.

Bias in subgroup analysis is less likely if the subgroups were defined before the reviewers selected the included studies. Cochrane systematic reviews generally are conducted this way.

8. How old is it? As new research is done, even systematic reviews become outdated. It’s hard to give a specific age that is “too old”, though, since some topics will have many new studies each year, and might be outdated at 18 months old while others rarely get anything new, and might still be relevant three years after publication.

9. How can I apply this? Again, this is always your last, but very important, consideration. Is this information that is applicable to your practice? To the population you serve? How can you best use this information?

Systemic reviews are among the best types of evidence out there, but only if they are done well. Read carefully, and consider the quality of the analysis as well as the quality of the included studies.

Next up: A roundup of other types of articles you may encounter in journals.

Uncategorized , , ,

Planned home birth and neonatal death: Who do we believe?

August 17th, 2010 by avatar

The (in)famous Wax home birth meta-analysis hit the scene over a month ago. But the buzz doesn’t seem to be dying down. In the weeks since the original pre-publication and press release, editors at The Lancet and BMJ have both weighed in, and there’s a steady stream of media attention. While all of the media have dutifully quoted midwives in leadership positions saying the meta-analysis is flawed (an assessment with which I agree), I still keep coming back to the question I asked in my earlier postdid we need a meta-analysis to establish the neonatal outcomes of planned home birth? We had, after all, a very large, methodologically rigorous study on home birth safety involving over a half million women that was published less than 2 years ago. Won’t that suffice?

I had a chance to interview two of the researchers who conducted that study when I was in Vancouver for the Normal Labour & Birth International Research Conference. Simone Buitendijk, MD, is Professor of Maternal and Child Health and Midwifery Studies at the Academic Medical Center of Amsterstam and heads up the Child Health Programme at the Netherlands Organisation for Applied Scientific Research. Ank de Jonge, the study’s lead author, is a practicing midwife with a PhD in public health who works at the Midwifery Science section within the EMGO Institute for Health and Care Research at VU University Medical Center in Amsterdam. I gained some new insights from them about their research and the Wax meta-analysis. Based on those interviews, and despite having written about the meta-analysis twice already, I thought it was time to ask anew: which is the “better” evidence for determining neonatal outcomes of planned home birth: the de Jonge cohort study or the Wax meta-analysis? Let’s have a look at some objective criteria and see how each study measures up.

Study size (home birth group):

  • Wax: 9,811
  • de Jonge: 321,307

That’s right, the Dutch neonatal mortality analysis is 33 times the size of the neonatal mortality meta-analysis. And believe it or not, this was BRAND NEW news to me that I didn’t realize until I spoke to de Jonge and Buitendijk. Although I had access to the full-text of the Wax meta-analysis and in fact looked critically at it (heck, I blogged about it!), I completely missed the fact that while the de Jonge study was “included” in the meta-analysis, it was excluded from the analysis of neonatal mortality, which was the major finding given so much attention by the media.  On the one hand, I’m pretty embarrassed to have made such a major error. On the other hand, it just underscores how misleading it can be for professionals or lay people to read headlines about a meta-analysis of “hundreds of thousands” of births finding triple the neonatal death rate.  Wax’s neonatal death data don’t come from hundreds of thousands of births at all. Not by a long shot.

Mechanism to ensure data were from planned home births:

  • Wax: mechanism varies across the included studies. In Pang et al., which contributed 63% of the home birth data and accounted for 12 of the 18 neonatal deaths in normally formed newborns, researchers relied on birth certificate data that did not differentiate between planned and unplanned home births, and assumed that any birth certificate for a baby born at home at or beyond 34 weeks, signed by a midwife, nurse, or doctor was a planned home birth, a method that has not been scientifically validated and has been widely criticized. Unplanned home births are riskier than planned home births with qualified attendants.
  • de Jonge: midwives routinely record the planned place of birth in a national perinatal database that covers 99% of births and is linked to another database of neonatal deaths by a validated method. Planned place of birth was unknown for 8.5% of the population, and the outcomes of this group were analyzed separately and reported.

Definition of neonatal death:

  • Wax: death of a live-born infant between 0 and 28 days
  • de Jonge: death of a live-born infant between 0 and 7 days (the World Health Organization definition of early neonatal death)

The appropriate definition of neonatal death has been a major bone of contention in the comments on this and other blogs that criticized the Wax meta-analysis.  Both 0-28 days (neonatal death) and 0-7 days (early neonatal death) are accepted definitions. Proponents of using early neonatal death argue that it is more sensitive to events occurring around the time of birth, such as hypoxic injury resulting from inadequate fetal monitoring or a sudden emergency like a cord prolapse or placental abruption. Indeed, some of the late (8-28 days) neonatal deaths reported in Wax resulted from sudden infant death syndrome, a condition that has nothing to do with planned place of birth. On the other hand, proponents of using 0-28 day mortality point out that some babies experiencing severe hypoxic injury in labor or birth may be kept alive for many days in a modern neonatal intensive care unit.  Their deaths should be counted as birth-related even if they don’t die as soon after birth.

Regardless of which is the more appropriate measure, I was shocked by something de Jonge and Buitendijk revealed in their interview. Wax never contacted them to ask for their 8-28 day mortality data. It is standard practice among researchers who conduct meta-analyses to contact the authors of the original papers to obtain unpublished data, clarify methodologies, or ask for data in a compatible format. One would think that if Wax was truly interested in whether planned home birth caused neonatal death up to 28 days, he would be very motivated to get his hands on the Dutch data set. And while de Jonge and Buitendijk told me that those data are not as complete as the early neonatal death data (because some pediatricians don’t reliably enter their patients’ data), they do in fact have the data up to 28 days and would have supplied it to Wax had he asked. Instead, they have done the analysis themselves and submitted it for peer review.  (Therefore, we’ll have to wait for the results.)

What were the characteristics of the population?

  • Wax – no requirements for home birth eligibility were defined for inclusion in the meta-analysis. Individual studies included in the meta-analysis varied in their mechanisms for determining eligibility. As noted above, the largest study that contributed the majority of neonatal deaths relied on birth certificates. Women with any of 18 medical conditions documented on the baby’s birth certificate were excluded. Neither the study authors nor Wax and colleagues comment on whether this is a reliable method for defining “low-risk”. (As someone who routinely completed birth certificates when I was practicing, my guess is that it isn’t.)
  • de Jonge – National guidelines (“Obstetric Indication List“) define who is eligible for primary midwifery care and home birth. These conservative guidelines ensure that the population of women having planned home births are healthy and at very low risk of complications.

The Dutch study has been criticized because it is, well, Dutch – midwifery and home birth in the Netherlands are highly regulated and integrated into the system, and there are clear eligibility guidelines. The same isn’t true of the United States, so we can’t generalize the results here or elsewhere where home birth is marginalized (e.g., Australia). What the Dutch study gives us, though, is a clear model to emulate in order to make sure home birth is as safe as it can be – regulate midwifery, provide continuity of care for women who need to be referred, and make sure only low-risk women are having home births. Instead of acknowledging this and moving forward to optimize safety, Wax and colleagues chose to mash together data from five different countries and four different decades with no attention paid to which women were and were not eligible and spit out an authoritative answer to the question, “Is home birth safe?” “Is home birth safe?” is a bogus question to which there is no answer. Context, training, system integration, and perhaps above all else the characteristics of the population matter. Any study worth its salt will describe these factors in as robust detail as is feasible. Combining and meta-analyzing data from dissimilar contexts may make sense in other areas of health care, but when context is everything, what’s there to gain?

A note about comments: please keep it civil and on point. I’m OK with debate, discussion, and disagreement. Name-calling, personal attacks, and other degrading commentary will be deleted or edited.

Uncategorized , , , ,

Meta-analysis: the wrong tool (wielded improperly)

July 10th, 2010 by avatar

A lot has been said about the new meta-analysis of home birth. (Here is an excellent summary from Jennifer Block.) Canadian physician Michael Klein has been widely quoted as saying that the meta-analysis, a potentially valuable statistical tool, was performed poorly because the researchers included studies using discredited methodology, as well as studies that are decades old. “Garbage in, garbage out.” I totally agree with this assessment. I also take issue with the fact that the researchers did not display the standard “forest plot” that customarily accompanies a meta-analysis to illustrate how the relative magnitude of observed differences in the individual studies and the pooled analysis. And I’m perplexed by the use of a fixed-effects model for the analysis of neonatal death.

But I want to take a step back and ask a larger question - is meta-analysis even appropriate for the study of home birth?

Meta-analysis is a statistical process that pools data from multiple studies. It is intended to achieve two related goals:

  • have adequate statistical power to detect differences in rare but clinically important outcomes (such as perinatal mortality among babies of healthy women)
  • establish a definitive answer to an important clinical question, so that policies and practices can adapt to conform to the new “truth” and other researchers don’t have to study the issue anymore.

Let’s look at these two issues separately in the context of the Wax meta-analysis.

Statistical Power

Lack of statistical power could not possibly be the rationale for conducting a meta-analysis on the safety of home birth. That’s because there already is a study large enough to detect differences in intrapartum and neonatal death. In fact, it contributed 94% of the data on planned home birth in the meta-analysis (321,307 of 342,056 planned home births). That study found virtually identical rates of neonatal death in both the planned home and planned hospital births*, with relatively narrow confidence intervals. Neonatal deaths on day 0-7 occurred in 3.4 per 10,000 of each group and when combined with intrapartum mortality and adjusted for confounding factors, the relative risk was 1.00 (95% CI 0.78 to 1.27). That means that there was a 95% likelihood that planned home birth results in somewhere between a 22% reduction and a 27% increase in intrapartum or neonatal mortality.)

By adding a bunch of smaller, older, and flawed studies, excluding the intrapartum deaths (which may be affected by intrapartum events and therefore are potentially modifiable by the birth setting) and adding deaths that occurred between 8-28 days (which are less likely to be related to intrapartum events and therefore are less modifiable by birth setting), we suddenly have nearly three times the neonatal mortality rate with planned home birth and a confidence interval you could drive a truck through?  (a 95% chance that home birth increases the risk of neonatal death by somewhere between 32% and 625%)  Hmmm…

Definitive “truth”

The other reason to undertake meta-analysis is to definitively settle a clinical question. Meta-analysis, after all, holds a privileged place atop the evidence pyramid, where it is considered the “best evidence.”  But is a deeply flawed meta-analysis really better than an adequately powered, methodologically sound study? The answer, of course, is no. All the meta-analysis does in such cases is separate the reader from the primary source of the data so that they can’t assess it for themselves, while putting the evidence-based stamp of approval on whatever statistics the meta-analysis software spits out. But people with a political motivation to authoritatively declare a certain definitive truth may realize that most people don’t bother to check to see if a meta-analysis is done appropriately or critically assess the quality of the included studies. They just go, “Oh look, there’s a meta-analysis of home birth and it said it’s 3 times riskier than hospital birth. That settles that! It’s a meta-analysis, after all!”

So if not a meta-analysis, then what?

OK, so if meta-analysis was not the right tool, what is?  And can we stop studying the safety of home birth now that we have that large study that contributed 94% of the home birth data to the meta-analysis?

The way I see it, the large study that showed equivalent perinatal outcomes between home and hospital birth tells us definitively that home birth can be safe. But it doesn’t tell us that home birth is intrinsically safe. We need to continue to study home birth using all of the tools in the research toolbox, qualitative and quantitative, to determine under what circumstances home birth is safe and how to optimize care and outcomes in all birth settings. And we need to stop pushing home birth underground in the United States where it remains a fringe alternative, poorly integrated with the maternity care system, with no standard safety net in place for women who begin labor with the intention to birth at home but turn out to need hospitalization in order to birth safely. Shame on the American Journal of Obstetrics and Gynecology for making this task even more difficult than it already was, by publishing and publicizing a junk meta-analysis.

*edited 7/12/2010 to correct a (serious) error. Sentence previously read “virtually identical rates of neonatal death in both the planned and unplanned home births.”

Uncategorized , , , ,