It is often said that randomized controlled studies (RCTs) are the gold standard in healthcare.
As David Shaywitz explains in a piece in Forbes Will Real World Performance Replace RCTs As Healthcare’s Most Important Standard?:
The value of RCTs lies in the random, generally blinded, allocation of patients to treatment or control group, an approach that when properly executed minimizes confounders (based on the presumption that any significant confounder would be randomly allocated as well), and enables researchers to discern the efficacy of the intervention (does it work better – or worse – than controls) and begin to evaluate the safety and side-effects.
But sometimes there is a mismatch between the results of clinical trials and actual real world experience. The intervention performs much better in scientific studies than in the general public. Why?
The subjects who enroll in clinical trials … may not be representative of either the larger population or of the patients who are likely to receive the intervention currently under study; groups underrepresented in clinical trials include the elderly, minorities, and those with poor performance status (the most debilitated).
This begins to get at what may be the most significant limitations of clinical trials: the ability to generalize results. The issue is that clinical trials, by design, are experiments, often high-stakes experiments from perspective of the subjects … as well as the sponsors, who often invest considerable time and capital in the trial. Clinical trial subjects tend to be showered with attention and followed with exceptional care, and study investigators generally do everything in their power to make sure subjects receive their therapy (whether experimental or control) and show up for their follow-up evaluations. Study personnel strive to be extremely responsive to questions and concerns raised by subjects.
But in real practice, YMMV, as they say on the interwebs — your mileage may vary; adherence is less certain, evaluation can be less systematic, and follow-up more sporadic…
That’s why real world evidence (RWE) is becoming increasingly important in assessing healthcare interventions.
Consider our experience with birth control methods. Early on it became apparent that birth control methods performed much better in scientific studies than in real world use. That’s because, depending on the difficulty of using the method, people don’t always use it properly or regularly or both.
This slide demonstrates the difference between the two:
How each method works in practice is much more important to the individual than how it works in theory. As a result, when providers discuss the risks and benefits of various forms of contraception, we use real world experience as the basis of our discussions.
The same principle applies even when randomization is not possible because it is unethical. Scientific studies are very valuable, but real world experience is possibly even more important.
Consider the case of homebirth. One of the most widely quoted papers on the topic is the Birthplace Study that found in a carefully selected population, subjected to much more stringent eligibility requirements than those in the real world, and followed much more carefully than real world patients, homebirth increased the risk of poor outcomes compared to hospital birth for first time mothers and was essentially equal for mothers having a second, third or higher order child. That’s good to know, but just as in the case of contraception, real world experience is more important than theoretical results.
So how do the outcome of homebirth and hospital birth compare in the real world? The British National Health Service almost certainly knows since they collect the data, but they won’t tell the public. I’m willing to bet that if the real world data showed homebirth in the UK to be as safe as hospital birth it would have been released years ago. The fact that it is not available suggests that in the real world homebirth, which the government has been aggressively supporting as a way of saving money, is significantly more dangerous than hospital birth.
Just as in the case of contraception, real world data is far more relevant and far more accurate than data from the best studies. We would never counsel patients on the risks and benefits various forms of birth control using theoretical effectiveness. Therefore, no one should be counseling women on the risks of homebirth using the Birthplace Study.
When it comes to breastfeeding, the dichotomy between scientific study results and real world experience is massive. The theoretical benefits of breastfeeding — based on extrapolation of small studies that assume causality — are large. The real world benefits of breastfeeding are almost non-existent. There is no real world evidence of which I am aware that shows that increasing breastfeeding rates saves any term babies or any money in actual practice. For example, there seems to have been no impact on infant mortality or healthcare expenditure despite the fact that the US breastfeeding rate has tripled in the past 45 years. Countries with the highest breastfeeding rates have the highest infant mortality rates while countries with the lowest infant mortality rates have the lowest breastfeeding rates.
What are we to think of situations in which the interested parties suppress real world evidence while aggressively promoting scientific studies? The drug Vioxx offers an instructive example. In scientific studies, Vioxx had tremendous benefits and few risks. But then reports began to come in to the manufacturer that in real world experience Vioxx increased the risk of heart attack and stroke. The manufacturer tried to suppress that evidence and many people were harmed as a result.
Homebirth is treated by its advocates the same way as Vioxx was treated by its manufacturer. Real world evidence of harm isn’t released to the public. Breastfeeding is treated by its advocates the same way Vioxx was treated by its manufacturer. Although the information on neonatal hypernatremic dehydration, kernicterus, infants smothering in or falling from their mothers’ hospital beds in the wake of closure of well baby nurseries is available from a variety of databases and published in scientific papers, lactation professionals simply ignore it. Over and over again they cite theoretical mathematical models created by Drs. Melissa Bartick and Alison Stuebe, and completely ignore both real world harms and lack of real world benefits.
So which is more important in health care: scientific studies or real world evidence?
Shaywitz offers his view:
…[A]t its best, real world evidence provides an opportunity to evaluate medical interventions on what arguably matters most – real world performance …
Real world evidence is not always available, but when it is — as in the case of homebirth and breastfeeding — it must take a central place, equal or superior to evidence from scientific studies. Partisans are going to resist, but that’s all the more reason to make sure that individual women have access to it. As with birth control methods, women can not make informed medical decisions in the absence of real world evidence.
https://slate.com/technology/2018/05/how-the-c-section-went-from-last-resort-to-overused.html
Beat me to it. How irresponsible! I like the tired twice-what-the-Who-recommends “fact.”
Not quite OT: My friend with the 30-weeker watched a lactation consultant on a TV programme or Youtube or something. She was very impressed with the claim that at the baby’s fourth day of life, ALL mothers want to die of the agony that milk coming in or changing or something caused. Except for just her, clearly, because she never felt such a thing. But rest assured, it was her fault and the fault of all other mothers that did not feel it either because they didn’t pump enough on the second day or something. It’s never a fault of the LC “science”. She enjoyed watching this ‘one man show” in the same manner that she enjoys watching a good comedy. Luckily for her, she wanted to breastfeed for herself, not to make her baby a superhuman, so she was perfectly OK with throwing the pump away and buying some formula. And of course, they’re vaccinating and all… although a friend of hers has already managed to send her a link about the harm of vaccines and the encouraging, “You still have the time to make a decision!”
Uuuuuuuugh so a high school acquaintance of mine recently had a homebirth and was sure to let Facebook know. She’s an otherwise intelligent woman, so the only thing I can figure is that she’s either fallen for the woo or trusts these bogus studies. It is so infuriating to see her brag about putting her child’s life in danger like this.
Yeah, an acquaintance of mine was positively smug talking about her homebirths attended by an LPN. However, she’s done with birth, so no more kids will be endangered by her foolishness.
This this this this this! I was talking with a friend yesterday who is pretty cluey on reading studies and understanding how they’ve been manipulated. One was posted showing homebirth outcomes as being the same as hospital, for low risk women. My friend pointed out, yeah but that’s a true low risk cohort, not what happens at real homebirths here! They’ll use that study to prove homebirth is safer (because of the lower morbidity) and therefore safe even with risk factors! Then I said, yeah, even low risk women can lose a baby though, like our other friend and she said, yeah, but that doesn’t go in the data because the midwife in charge sold her up the river claiming she intentionally freebirthed. So you have to realize that often the data set they are using has excluded bad outcomes that were 100% avoidable had the baby been in hospital and the final numbers are so small that one or two births excluded from the data has a big impact. If we tally up all the homebirth deaths we know of many of them would not be included in that data set because the woman had a VBAC, or she was over 35, or she refused a GTT, or she went over 41weeks or she was a little overweight. All things homebirth advocates claim are completely safe and are allowed in Australia with a registered midwife!
OT: One of the moms in my postpartum support group is pretty far into the woo, taking her baby to a chiropractor, getting tongue and lip ties clipped, that sort of thing. The kid seems to have some legit food sensitivities, although she’s growing well, and the mom has been stressing that they can’t figure it out. Today the group leader was trying to reassure the mom that it will be OK, which prompted the mom to break down sobbing that she’d read her kid would get “ADHD and autism” because of these food sensitivities.
I wanted so bad to give her a hug and say, “Honey, don’t worry, those blogs you’ve been reading are total bullshit.” I couldn’t really do that but I tried to make some general noises in that direction. Now I suspect she thinks I was just blowing off her concerns. Sigh.
The woo’s have taken it beyond vaccines and now claim that food sensitivities/intolerances cause autism and adhd?! I know I’m asking for way too much rationale from them here, but by what mechanism? Just, how? How can anyone believe that? Sounds like this group needs to be much more vocal in an objective, scientific stance as to help those struggling with irrational fears. Why can’t you tell her she’s been exposed to total shit and how to find facts which will calm her concerns? This is just sad 🙁
*something something gluten something something toxins mumble mumble gut health* There’s a sizable quack contingent that believes autism is a gut disorder. And people have been blaming ADHD on dyes and additives for a while.
As for this group, I don’t know what to do. There’s a lot of less terrible woo floating around that people just accept (it doesn’t help that the leader is a doula).
Completely OT but I kind of need to vent here–horrid outcome after home birth, though I don’t know how much the location contributed. Young couple with a child about six or so. This pregnancy was described as perfectly healthy and normal, mother went into labor at 37 weeks, baby girl was born at 7.5 pounds. Almost immediately baby had severe respiratory distress and went to hospital by ambulance. Father followed ambulance; hospital staff trying to stabilize baby, having very little luck. Then mother, still at home, develops severe PPH, had some retained fragments of placenta. She gets sent to hospital by ambulance (luckily, same hospital at least). Mother requires removal of placental fragments and a transfusion. Next day they receive diagnosis for their daughter: autosomal recessive polycystic kidney disease, kidneys enlarged, lungs full of fluid, heart affected. At 4 days, they remove life support and baby dies. Parents are now $50K in debt besides having lost their much-longed-for baby. My question: could the baby’s kidney problems have been spotted on a prenatal ultrasound? If there could have been warning, it just seems at least that might have given the parents a chance to prepare themselves, to make decisions about care, plan for who they needed to have at the birth, any rituals they needed, possibly even forego the four days on life support–the whole situation of thinking everything was perfectly healthy and normal and then having it all go south as soon as the baby was born is a big source of suffering, plus I hate to think of the little baby, unable to breathe until an ambulance arrived and took her to the hospital, and then the mom with the hemorrhage which has to be terrifying even if you know what’s happening. Does the “autosomal recessive” portion of the diagnosis make it harder to diagnose? Just trying to sort things in my own mind. Grandparents are old friends of mine. Whole family is absolutely shattered, and then the huge bill on top of it.
I am so sorry for your loss, and theirs. What an absolutely horrible situation!!
Not a doctor, but my mother-in-law has polycystic kidney disease, and it is a pretty awful condition. A small upside is that hers is just a fluke of genetics and not the inheritable type. Regardless, everyone in the family dutifully notified their medical team with each pregnancy and we all had extra attention paid to the developing kidneys during ultrasounds. Therefore, it probably would have been caught prior to birth if she had had regular ultrasounds.
PKD is surprisingly common. Unfortunately, it makes up a large proportion of the diagnoses on our transplant list…but once they get a transplant, the disease doesn’t recur in the new kidney, so they generally do fine.
That is something I didn’t even think to look up, so thank you for that information! My mother-in-law finally received a (deceased) donor kidney a few weeks ago after being on haemodialysis for ~9 years. It took a while to get to this point, but everything is going really well and we’re cautiously optimistic. She’s certainly looking much better now!
Oh wow, that’s amazing! Yes, with some genetic disorders that are more metabolically-based, the disease will recur in the transplanted kidney, but with PKD that’s not an issue. Good luck to her!
Thanks! We’re all very excited for her. She has been really lucky – it was a great match, the kidney started working right away, and she hasn’t needed any dialysis to help it cope. The only complaint she has had was about getting used to urinating again (and having to measure/log it all)
Perhaps the autosomal recessive version is more rare, or the family may be making meaning out of thinking it is, because they described her disease as “extremely rare” and I thought PKD was not extremely rare. They may be constructing a “would have been the same in the hospital” narrative, I don’t know. They will never be the same after this, I imagine, but I hope some more balanced view of alternative vs. medical treatment is an outcome. (she said, trying to be tactful)
I think lay people mean “rare” to mean “I’ve never heard of it”. While doctors use rare to mean “The best specialists in the country see a few of these in their whole career”.
It is of course likely true that the midwife has never heard of it, and it is in her best interest to make it seem as unforseeable as she can.
Sometimes, you do everything “right” and get unlucky. These parents might be trying to blame this on bad luck, which was part of the trouble, but not all of it.
I think it is probably the “would have been the same outcome even if we had been in the hospital/had appropriate prenatal care/etc” because they’re grieving, and the truth – that this did not necessarily have to happen like this – is just too painful.
Thanks. I am glad for your mother-in-law finally getting a kidney, but what a long wait and a hard slog.
“Autosomal recessive” just means that the parents each had one healthy version, and one messed up version of the gene in question, but the one healthy version was enough to make them healthy. Poor baby had the bad luck of getting a bad copy of that gene from each parent. 1 in 4 chance.
The 6 year old needs to be checked, she has a 2/3rds chance of being a healthy carrier.
According to the interent, yes, songraphy has likely been a major contributer to better survival rates of babies with this disease.
https://pkdcure.org/what-is-pkd/arpkd/
Thanks for the info, it may be that at some point I can suggest testing for the older child, and appropriate care and followup should the couple decide to try for another pregnancy at some point. And, yeah, you would think they could spot the incomplete placenta, but I wasn’t there, so…
I could have happened that the midwife was so taken up with dealing with the baby that she didn’t have a chance to properly assess the mother…yet another reason why homebirthing is bad…I imagine at a hospital, it is someone’s job to ignore the fuss around the baby and make sure the mother is okay.
Having a homebirth didn’t help the situation. Adequate prenatal care including an ultrasound might have prevented some of the related issues with lung and heart development.
Fetal lung development requires a certain amount of amniotic fluid. Amniotic fluid is mostly produced from the liquid that passes through the baby’s kidneys after entering the baby through the umbilical cord. Since the baby’s kidneys were compromised before birth, the little pumpkin wasn’t producing enough amniotic fluid which prevented her lungs from forming. (That’s part of the reason OB/GYN take those rather uncomfortable height of the uterus measurements during pregnancy. Too little – or too much- amniotic fluid can lead to a uterus that the “wrong size” for the mom’s due date.)
A baby who is born with immature or abnormal lungs can sometimes benefit from immediate breathing support like extra oxygen or having someone use a non-invasive positive pressure breathing apparatus (like CPAP) or being put on a traditional ventilator or a high frequency ventilator or having surfactant added to their lungs. All of these things require a trained RN or respiratory therapist – and most of them require multiple trained people to do quickly and effectively.
In the absence of a trained resuscitation team, the kiddo couldn’t breathe well – and the oxygen deficit she was piling up was doing more damage to her lungs as well as her kidneys. Once she got breathing, the severely damaged lungs cause issues with the blood pressure flowing back into the heart – and that can cause severe heart issues quickly.
My heart goes out to you, her grandparents, her parents and her sibling. They’re in a shitty situation – and the type of situation that I am fearful of every time I meet someone who is concerned about the effects that ultrasounds might have on their baby…..
RIGHT??? It’s a true back-to-nature situation, and nature can be a bitch. As with my friend who had end-stage achalasia because he always interpreted his swallowing difficulties as something spiritual in nature. Some people would rather die than be ordinary.
Agree with Mel. Ultrasound would have shown wonky amniotic fluid levels early on, and then probably a lack of “practice breathing”. This would have prompted investigation of the kidneys, etc. This condition is readily diagnosed via ultrasound and a host of measures are then put in place for the delivery.
Mom had no prenatal ultrasounds?
Thanks for the information. That’s helpful for me to know, so I can stay out of some of the drama, right? I am guessing no ultrasounds.
Autosomal recessive just means that it’s possible for the baby to be affected even if neither parent knew they were a carrier. (This isn’t even tested for in prenatal screening). More common is autosomal dominant PKD, which generally develops later in life.
Agree, problems with baby’s kidneys would be easily seen on an ultrasound. This may even have been correctable surgically before the baby was born. I suspect this mom declined prenatal ultrasounds. Very sad…
Thanks for the info. Makes it sadder. Yes, at birth scans showed enlargement of the kidneys and cysts present in both, so I guess you’re right, they could have seen it. You’re probably right about the ultrasounds. Trusting birth and all that.
I’m so sorry for them. That’s terrible.
Do they not have health insurance? Why the $50k bill?
Probably crappy insurance and probably paid for the midwife out of pocket. I don’t know really, but that’s my guess.
Replying to self: they may have had insurance with very limited coverage for pregnancy/childbirth. On the plus side, the family has expressed deep appreciation for the care provided once they got to the hospital. May their eyes be opened. Poor NICU team though.
I have to ask, says who? Is this the assessment of a doctor, doing full prenatal care? A midwife? The mother?
As others have noted below, without proper screening, how can you know it’s “perfectly healthy”?
I wondered that too. If no tests are done to pick up any issues, then in the absence of something serious to indicate a problem (massive retention of water, severe pain, bleeding, no fetal movement) there are about a million things that can be wrong with the baby that won’t show up until after they are born. There are a ton of things that are done now to fix problems in the womb that couldn’t be done in the past because there were no outward signs that something was wrong. I’m not heartless, losing a child is horrible.
I agree, it is awful that they lost their child, but I really think the narrative changes if it starts out, “They didn’t have any prenatal checkups and didn’t know if the pregnancy was healthy or not.”
I don’t know if that’s actually the case, and maybe they did have proper prenatal care. You just can’t tell it from this description.
Exactly, I only have the family’s description to go on, but Colorado homebirth midwives have a desperately shitty record. Another friend’s daughter had a home birth which resulted in a torn umbilical cord due to a velamentous insertion; luckily they had clamped before the placenta fully detached and the baby didn’t bleed, but the mom had a hemorrhage and retained placenta, had to go emergently to the hospital; the couple, encouraged by their midwife, felt the whole thing went well and was glad of the homebirth.
God, I never trusted that my child would be born alive and healthy. Mom’s stillbirths made an impression on my mind.
RIGHT???? Yeah, my suspicions are high. Certainly the family, and almost certainly the home birth midwife, described the pregnancy as “perfectly normal and healthy.” It’s so sad.
A lot of homebirths the Mothers have little or no testing. Women do their “research” online using the bullshit quacks their midwifery and homebirth pages link to – so they believe scans are dangerous, GTT is just a way to force women into hospitals, no such thing as overdue, breech is a variation of normal, GBS can be cured with garlic – and the midwives see themselves as having no responsibility to correct this (if they don’t actually believe it themselves). I’ve seen it time and time again, midwives believing their role is to “support” the woman. So yeah, often they think their pregnancy and birth was perfectly normal – and they may have been degrees away from tragedy – but as long as everything turns out – even if it includes a stay in NICU – as long as the baby lives – it was a success! If the baby doesn’t live – delete and ban.
Oh yes, horrid and tragic, and I suppose they’ll always wonder if this was preventable.
So sorry for the parents and the family. And you.
Thanks–it’s the family, really–couple generations of alternative lifestyle lovers, and this is not the way you hope they discover the weaknesses of that viewpoint.
To me, this question is akin to, “What’s more useful–a hammer or a screwdriver?” Well-designed RCTs are very good at determining whether an intervention is or is not efficacious, and it’s easier to control for confounders than in a cohort study or whatever. But yes, nothing works as well in real practice as it does in your nice, tidy study, so you do need studies of real-world effectiveness. (Also, RCTs aren’t great at picking up rare side effects, because of their small numbers.) I would say you need both to assess the value of an intervention.
Regarding breastfeeding, I don’t think you could do a proper RCT, but what you have instead are observational studies riddled with confounders (the best of which show little benefit for breastfeeding), and crappy theoretical models that posit huge numbers of lives and dollars saved. Eye-roll.
To me, it’s even a little more subtle. The difference between ‘ideal’ and ‘real world’ tells you things. In the case of birth control, it tells you how easy the method is – the difference for NFP and withdrawal is massive, it’s lower with the pill, and it’s lowest with implants. That doesn’t tell you which one is better for any given individual, but it helps to know that one’s personality and life constrains makes one a better or worse fit for a given method, and it’s a good guide for public policy.
This also holds for other drugs. There’s a well-known phenomenon that even the control group in a trial for an autoimmune/inflammatory condition will generally get better, and a lot of it is because when people are on a trial, they act better. They do all of the little lifestyle things that they just may not have the brain-space to keep prioritized normally. And the fact that the population as a whole falls off of those in ‘real-world’ conditions tells you how generally sustainable they are.
It can tell you other things about other drugs – how heterogeneous the disease is, for instance. You can carefully define your stage III/IV biopsy-confirmed LN population in a trial and get your efficacy numbers, but in the real world, doctors are going to give that drug to everyone who seems LN-y, and the efficacy will accordingly take a hit.
And these all apply to breastfeeding, IMO? It takes time and effort, and it’s heterogeneous. The results from formula will be more consistent, a la implants. That doesn’t mean formula is necessarily better, it’s just an important factor to consider.
Sometimes, you need both tools anyway. Just put together a toy box for the spawn and we even had to use both a wrench and a screwdriver at the same time.
They were so helpful, too. On the positive side, the spend the rest of the day playing “pite! yo, yo, yo!” as the shorter one said. Well a child’s toy box is definitely their treasure chest.
WAIT! Spawn is talking?!?!?!? How did he get so big? He was just born like last month or something, or so it seems.
You’re thinking of Mel’s spawn. Though he might be starting to babble. My spawn are 4 and almost 2. Kid2 is definitely talking, when she can get a word in edgewise
So I was really tired last night. Hopefully today my brain will work better.