Years ago, when I was working for a large health maintenance organization, I received a curious letter. It had come to their attention that my forceps rate (0%) was well below the average for obstetricians in our institution. I pointed out that my C-section rate was only 16% and I hadn’t left a single baby inside a mother. I asked if this were a problem? No one seemed to know.
At another point I got a notification that I had ordered more ultrasounds for my OB and GYN patients than average for that month. This time I asked whether any of the ultrasounds had been unnecessary? No one seemed to know … or care. It was just something they were required to measure.
[pullquote align=”right” cite=”” link=”” color=”” class=”” size=””]The composite index DOESN’T tell us whether an individual hospital is a safe place to give birth.[/pullquote]
I mention these anecdotes because they illustrate the dangers of simply compiling statistics and comparing them. My employers wanted to know if I was delivering patients appropriately, but my forceps rate couldn’t tell them that. They wanted to know if I was ordering unnecessary ultrasounds but comparing my ordering rate to my colleagues in a given month couldn’t tell them that, either.
It is important to measure, but you have to be sure you are measuring the right thing in the right way.
That was my concern when I read the breathless report in USA Today, The secret number maternity hospitals don’t want you to know, and why we’re revealing it.
It is extremely important to measure maternal complication rates. But are the people at USA Today measuring the right thing in the right way?
It’s not clear to me that they are.
According to the article:
[I]n the United States – the most dangerous country in the developed world to give birth – maternity hospitals’ childbirth complication rates are a well-guarded secret.
Many hospitals know them. So do many state health agencies, insurance companies and researchers. But they fear the complication rates are too complex for regular folks to understand.
In truth, no one knows what they mean.
There are certainly questions about the data. Some hospitals are more likely to treat patients with health problems, so childbirth complication rates are difficult to compare. Some of the best-equipped hospitals in the country may have higher complication rates because mothers who are very sick get referred there.
But there’s more to it than that: How useful is the method they used, a “composite index”? How valuable is it to compare one hospital to another or any hospital to a mean value? What is it we really want to know and do the measurements chosen by USA Today tell us what we want to know?
We want to know several things that are extremely difficult to measure:
Is the complication rate for a given mix of patients appropriate?
Has every complication that could have been avoided been avoided?
Has the hospital staff caused complications?
And were the complications that occurred appropriately treated?
Where did the USA Today “secret number” come from?
More than a decade ago, the U.S. Centers for Disease Control and Prevention created a method for calculating how often women giving birth endure severe complications using diagnosis and procedure codes that hospitals record in patient billing records.
The resulting “severe maternal morbidity rate” is like a composite score of things that can go wrong at the hospital before, during or after delivery – heart attacks, strokes, blood transfusions, hysterectomies and other emergencies that can permanently harm or even kill a new mother.
The first problem is the inclusion of blood transfusions (often a minor complication) with far more serious complications.
[T]he CDC method uses blood transfusions as an indicator that a woman may have hemorrhaged. But it’s impossible from billing data to know whether the woman received one unit or many units of blood.
Some experts say that could inflate the rate. Others note that transfusions are counted for all hospitals and a blood transfusion is not part of a routine childbirth.
No it’s not routine, but it’s also not an indicator of severe complications. It would be far better to look at the rate for women who got 3 units of blood or more. A transfusion that large is a good indicator of a severe complication. So right off the bat, the composite index is almost certainly inaccurate in modeling severe complications.
The next problem: in order for any measurement to have meaning, we must adjust case mix so we are comparing like to like:
[S]ome hospitals have higher rates because they are specialty facilities where doctors send the riskiest cases. Some serve more poor mothers, who often get less consistent prenatal care. Others serve larger numbers of black mothers, who tend to have higher rates of certain health problems, such as hypertension, that can lead to serious complications.
What about just looking at the outliers? The article is accompanied by this graph:
Complication rates are distributed in a bell curve (a “normal distribution”) with a very long tail. (You can’t really see that because of the editorial decision to lump all hospitals over 5% together creating a spike at the end.) A normal distribution with a long tail has particular implications in public health.
As the authors of The “Long Tail” and Public Health: New Thinking for Addressing Health Disparities explain:
The prevailing approaches to improving population health emphasize “shifting the mean” through prevention efforts that target large groups at high risk or through mass environmental control interventions that encourage small but universal changes in individual behavior. This approach has led to the search for “blockbuster” public health interventions that can have the largest effects on determinants of population health and individual behavior…
In a compelling critique, Frohlich and Potvin argue that the prevailing population approach may have the unintended consequence of exacerbating health disparities. Disease risk, they point out, varies not just by behavioral risk factors but also by socially defined groups that vary in their exposure to fundamental risks, for example, low education and low socioeconomic status. Broadly targeted population interventions that focus primarily on behavioral determinants may not be as effective under these conditions or with these groups…
And that’s why simply comparing the composite index of one hospital to the composite index of another hospitals is not particularly helpful. Any hospital that is within the bell curve has an acceptable complication rate. Most hospitals in the long tail probably serve vulnerable populations and their complication rates may be more closely tied to socio-economic factors than to hospital competence.
But that’s not particularly eye catching when compared to a “secret number that hospitals don’t want you to know.”
So what does the USA Today composite index tell readers?
Not much that is useful for them.
No one should decide where to give birth based on the composite index since direct comparisons aren’t really possible and all hospitals in the normal distribution are probably as safe as the others in the normal distribution.
The hospitals in the long tail very likely serve high risk populations and the problem is not necessarily that they are providing poor care; they may be providing exactly the same care as the hospitals in the normal distribution but that isn’t enough for vulnerable populations.
The USA Today composite index can highlight facilities that deserve additional scrutiny but unfortunately it the CAN’T tell us whether an individual hospital is a safe place to give birth.
Some years ago I worked as a midwifery consultant for an Israeli hi-tech startup which was building programs for the computers in US delivery rooms. Our first site for beta testing was a hospital which did a LOT of deliveries, but was definitely for a low risk population. It quickly became apparent that we had to find an additional site — a hospitalwhich specialized only in high risk — the higher, the better, to see if our program was functioning properly. Site A had nearly no complications whatsoever, while EVERYONE at Site B had lots of them. By the standards of this kind of study, Site B would be regarded as a worse hospital than Site A, while in fact it is one of the best in the US.
TheUS’s statistics on maternal mortality and morbidity are due to many factors, some of which have nothing to do with hospitals at all. A black woman who did not have antenatal care is a higher risk, even if healthy, than a white woman who did have full antenal care.
I recall something like this awhile back, where one local hospital was pointed out to have a much higher rate of CS and maternal complications compared to the other local hospitals. The hospital put out a public statement that boiled down to “no shit, dummies, we have the only maternal ICU within hundreds of miles. All those other hospitals send their hard cases here.”
The local hospital of my region has an extremely low rate of birth before 33 weeks.
Now you could use this data to pretend that they are very good at preventing prematurity. Or maybe it’s because any woman who shows up in labour before 34 weeks is air-lifted to an outside hospital with a pediatric ICU equiped to deal with severe prematurity.
And we are probably causing them to have less good stats by sending all our worst cases to them.
that stock photo is totally distracting me. Why are is the right side of her mouth a different color from the left? Am I the only person who worries about this stuff?
Why are her nails that ridiculous length? Why does her nail polish (perfectly nice on its own) not match or pop or tone with either lip colour?
No, you aren’t the only one.
She’s obviously not a health care employee-we aren’t allowed to wear long nails or acrylic falsies, its an infection risk.
I think it’s just the shadow of her finger that is making one side of her lips darker.
We have a very similar situation here in the UK. The NHS produces ‘league tables’ covering most specialities:
https://www.nhs.uk/service-search/performance-indicators/consultants/consultants-colorectal-surgery?ResultsViewId=1030
I choose colorectal surgeons at random as an example. You can search by region, by speciality etc and its supposed to be part of the open and transparent culture for patients to help them choose. The outcomes measured are survival rates, length of hospital stay, number of operations carried out by that surgeon (or whoever), and whether there were repeat operations.
But unless you have a very good grasp of what its measuring, it doesn’t really help. It’s crude and can be misleading, and blunt figures about readmission rates and complication rates are no good unless you know you are comparing like with like. These crude outcome figures don’t really tell you about the actual patients. Some centres specialise in a certain procedure, some centres act as centres of excellence and will look after very complex patients referred from less specialist centres, and in this cohort of patients the likelihood of complications is higher. It sounds awful, but if a patient dies, it doesn’t necessarily mean the surgery was carried out inappropriately or incorrectly. There’s a morbidity and mortality review of any patient who dies in hospital, especially those who die after surgery, but just recording the crude death rate of a particular surgeon doesn’t tell you much about their performance, and I don’t think crude figures can get those nuances across.
I would’ve thought the most experienced surgeons would get more of the sickest and most complex patients.
That’s the way it works generally-everything is getting super-specialized. When I was a medical student in the 80s, most standard hospitals (district general hospitals) had general surgeons. On a typical surgical list there might be a breast surgery, followed by gallbladder surgery, followed by someone getting their varicose veins out etc, and the same surgeon would do them all.
The outcome for patients is far better with more specialized treatment, so nowadays all breast surgery is carried out by breast surgeons, who do nothing but breast, and colon surgery is done by colon surgeons. And even within a speciality, there are areas of expertise-some intestinal surgeons specialize in the upper end of the GI tract, and others exclusively may do the bottom end. So a complex patient with more specialized needs would be referred to the more specialized surgeons, but obviously the more complex cases have more complications. It means that the outcomes for the very best super-specialized surgeons might look poorer than their less specialized colleagues which seems nonsensical, but only if you don’t take into account the pre-operative morbidity and complexity of that cohort of patients.
The medical profession is generally one huge family, and we all know who’s who in our own areas. In my little area of paediatric pathology, f I get a weird looking tumour, I know colleagues in other hospitals with specialist expertise in kidneys, or soft tissues, or liver tumours, so I can get a second opinion if I need to.
Ahhhh, welcome to the bullshit teachers have been dealing with for decades. My teaching career was spent in a high poverty urban school with a very high population of first and second generation immigrants. We were a “failing” school. You see, our students, many of whom came to us speaking no English and not literate in their native languages, and having never seen the inside of a school building before arriving in the US, were held to the same standards as the schools in the upper middle class neighborhoods. And while on the surface that seems like a good thing, it’s just not possible. I cannot take a 12 year old who’s not literate in their native language and can barely do more than basic addition and subtraction and have them performing on a 7th grade level as determined by a test taken in English 9 months later. My non immigrant students were dealing with extreme poverty. Many of my students were lucky to have one set of school clothes that got washed once a week. They came to school every day because they’d get fed there, and that was often all they ate. They came in on the weekends because my principal would flat out make up reasons to open to school to give an excuse to provide lunch. How can you learn math, or anything when you’re hungry? Every year some small child would get killed in this area of town just playing with toys on their front steps, or in once case, watching cartoons in their living room. Stray bullets from gang violence was an ever present threat, and innocent children died. How can you focus on literature when you’re constantly scared you wont’ even survive the walk home from school? Most of my parents were good people who were trying their best, but crushing poverty doesn’t allow for private tutors and after school activities, to supplement their children’s educations, like those parents in high SES neighborhoods. They want to be involved but it’s difficult when you’re working three minimum wage part time jobs just to keep a roof over your kids’ heads. My most memorable parent teacher conference was at midnight at a strip club, where the parent was a cocktail waitress, because that’s when she had some time to meet with me. And then there’s the parents who are too embarrassed to come in because they think they’re too dumb to be in a school. They’re not, but they’re afraid.
And yet, despite all of that, my colleagues and I could take those kids that just arrived in the US and get them reading and understanding math and science on maybe a 4th or 5th grade level. We could take the kid who came to us on a 3rd grade level and get them performing on a 5th or 6th grade level, all in one year. But even though we were advancing kids multiple grade levels in one year, we were failures. One year we all got fired and had to reapply for our jobs. Only half of us got our jobs back. They brought in “good” teachers from the high SES suburbs. It was, as any reasonable person would expect, an enormous disaster. Test scores fell. Attendance fell. Behavior referrals and arrests on campus rose. Gang involvement among the students outside of class rose. The school hadn’t completely recovered even when I left, several years later.
So now medicine, like education, is being treated to the “one size fits all” model, where bureaucrats who know nothing about the actual situation attempt to tell you how to do your jobs, much to the detriment of your patients. Doctors are suffering the same type of smear campaign teachers did, and are still dealing with. I wish I could say that education eventually won in the end, but 20 years after I first stepped foot in a classroom, the situation has only gotten worse. I truly hope that y’all have a better outcome than we did.
Those children are lucky to have such dedicated teachers, even though you are managed by a bunch of idiots. Thank you for sticking with them-I think education is the single biggest factor that is going to get those kids out of poverty. Its over 30 years since I left high school and I still remember my teachers (high school in the UK is age 11-18, you do your GCSEs at 16 (general certificate of education, used to be called O-ordinary-levels), your A levels at 18 and then go onto university). I was at a comprehensive school-a state school, not private (in the UK private schools where you pay for your education are called ‘public schools’, I know it doesn’t make sense). It was an OK school, it tended to turn out practical people-we had a school farm and it offered farming qualifications, and also did trades based vocational qualifications like City and Guilds courses, as well as the standard academic subjects. I had some absolutely amazing teachers-dedicated, passionate, ridiculously hard working folk who simply wanted their kids to do the best they could. It’s disgraceful that they struggle to get the funds and resources to do the job properly-I’ve a couple of friends who are teachers (both primary school-that’s 5-11 years) and I know they spend a lot of their own money buying supplies because the school can’t provide them.
Oh, BeatriceC, I feel like we are long-lost sisters!
I’ve been out of K-12 education for around 4 years now and just started subbing again. I enjoy subbing in the two districts I taught in plus the two other nearest districts which have similar demographics which is a mix of very recent immigrants, refugees, school-of-choice families from the large city near us and families who have recently arrived in the middle class.
It turns out I can handle classrooms very well. Even the class of junior high students whom 8 separate teachers (not exaggerating at all) warned me were a rough group. They weren’t particularly rough – just very high energy, very chatty and needed redirection – or in other words….junior high students.
After a few “miraculous” management times like that, someone would stop me and ask why I was teaching full-time anymore. I usually said something about a medically complicated toddler – but if I knew the person well I’d say, “Well, having four preps a day that I was creating/finding leveled content readings ranging from high elementary through HS was tiring. Having $50.00 a year for supplies for my 5 classes of 35 students each was exasperating. Seeing one or two students a year die or end up in prison for a bad crime they committed hurt. Being told that I was failing the students by not being able to get teens who had severely disrupted educations, were still learning english, were facing severe poverty or dysfunctional families to do equally well on the ACT/MME as the wealthy white kids I went to school with…that’s what broke me.”
We really do have a lot of similarities. I also tend to give the excuse of medically complicated kids as the reason why I no longer teach. That did play a role, but I am absolutely in a position now where I could go back if I wanted to. I just flat out don’t want to for all those reasons you state. Instead I sponge of MrC’s retirement accounts to supplement my own and volunteer my time tutoring refugee kids who are struggling.
I just couldn’t and I’ve never had a class of my own
I saw it too from a regional substitute viewpoint. One town not only had one of the poorest zips in the state in attendance but they also had a particularly good special ed program so people sometimes moved there for the better school. When you have a 16yo still struggling to learn kindergarten work, she’s not going to make grade level any
Some seven years ago, my mom got a new (private) student. A kid who had recently come from China and couldn’t speak Bulgarian OR Chinese. I remember meeting him after about a month of lessons (in Bulgarian. The purpose was to get him to understand what was being said to him in the Bulgarian school.) He had been to the mall and wanted to ask what Santa Claus and all this was but he couldn’t. He didn’t have the words. Fortunately, he’s quite talented at drawing, so he made a drawing of what he wanted to know and my mom started explaining. I remember thinking, “She’s never going to pull this off. NEVER.” Our textbooks are hard even for us. Honestly, I think they’ve been designed to the purpose of getting kids disgusted with learning. What chance did he have? It turned out, quite a big one. He stayed before or after lessons with my mom and dad and talked to them, watched the ways of a typical Bulgarian family, ate Bulgarian food. My mother read all his lessons, in all subjects, with him. The first words he told me when he could already speak were that he was so very unhappy and he wanted to go back to China.
Today, he’s well adjusted. He speaks Bulgarian and English like every student here – well, his Bulgarian is not always correct because, honestly, it’s hell to learn even for many of us. But he speaks the way his classmate speak – word, phrases. He has friends. And he no longer wants to live in China. He credits my mother for his success. She’s very happy and proud. She prides herself on his being able to communicate more than she does on all the students she has helped to reach some really, really outstanding academic results. Because with him, she put forth much more effort.
And people say *I*’m the smart one because I am not “a mere teacher”. Shaking my head.
This is a ‘solution’ to the ‘problem’ of people trying to evaluate things they don’t have the background to understand and evaluate, so they generate a ‘surrogate marker’ that is ‘supposed’ to represent what it is they can’t evaluate.
Another extreme case of this is using the ‘citation index’ to evaluate the research career and ‘productivity’ of a scientific researcher. How many other researchers cite journal articles in the journal that a particular researcher also published in, is a strange metric to even consider might be relevant. The only reason I can think of, why this is considered a ‘relevant’ metric, is because people using this metric don’t have the expertise to actually read what the researcher has published, understand it, and then evaluate it in the context of the scientific field at the time (the only ‘actual’ metric that makes sense).
Insurance companies and non-MD administrators can’t look at a patient’s chart, and evaluate the ‘quality of care’ that that particular patient received from a particular provider, so they generate these ‘surrogate measures’, which can be worse than useless.
When I was chair of our community hospital’s OB department I was constantly beating back similarly useless information. At one point one of our doctors got flagged for having excess incidence of injury to innocent organs. On inquiring further I was told he had ‘too many’ third-degree lacerations. I replied that he also had one of the lowest c/s rates, and that I suspected the two measures might be related. After some discussion the matter was settled when we agreed that the anal sphincter is not an innocent organ.
Another time, our repeat c/s rate was felt to be excessive by the risk manager. We looked at the situation and discovered that a lot of women were delivering their first babies at the local catholic hospital and then coming for their LAST babies to our hospital so they could have their tubes tied along with their c/s. More useless numbers.
We recently got an award for 100% breastfeeding, and I was astounded because we have steadfastly refused to adopt BFHI and many of my patients have shared that they do no want to breastfeed, and when I asked about THAT, the nurse manager said they had studied the information request carefully and realized that they could reply, in all honesty, that 100% of women who stated they wanted to breastfeed left the hospital breastfeeding. Sometimes with supplementation, but still. So there.
When I was teaching, our school always went through the yearly drama of setting academic goals. I like the idea of academic goals for students – but in education, there is no reward for hitting internal goals while if you miss the internal goal the district might decide to penalize your school somehow.
Pretty sure that’s the absolute opposite of the mentality that leads to honest goal setting, but I digress.
The trick I passed on to other teachers who were not in science or math was that the administration liked percentages but never really got how percentages worked in real life. IOW, stating that we going to raise our average ACT/MME math score by 30% this year sounded like a high-performing, stringent type goal. When you did the math, that meant that we would move our average score of 2 (when a 19 was average for a college bound senior) to 2.6.
It’s so similar to this study by one of our local Brisbane unis (QUT) last year — one of the main “risk factors” for giving birth by C-section and not “normally” (unmedicated vaginal birth, their wording!) was being in the care of a private-practice obstetrician, as opposed to a dedicated private midwife.
How shockingly surprising, when you think about how the respective providers’ patient populations self-select to be able to easily access exactly these respective birth modalities. I know I went private OB because hell yeah elective CS! And in contrast, a friend went private midwife because she wanted a home birth with pool and chanting.
But of course a variable like patient preference or intended mode of delivery was not even measured in the questionnaires that the study’s data was based on…
I think that “do patients who want to breastfeed get supported to breastfeed” is a much better measure than “do patients get supported to breastfeed” since, well, some won’t want to and shouldn’t be pushed to.
I know so many women who started breastfeeding in hospital and dropped it when they went home simply to avoid the formula argument.
Personally, I’d be less concerned about giving birth in a hospital where there was a 5% complication rate than one of those hospitals with a 0.1% complication rate if that’s all the data I have available.
Yeah, I get that there was a push to pre-determine the risk factors for pregnant moms and send them to hospitals accordingly – but that doesn’t do much good when a low-risk mom develops cardiomyopathy or starts bleeding massively after birth.
I’d prefer to be somewhere where there are lots of staff members who have seen and treated whatever I’m dealing with rather than being the first person who has had this complication this decade.
Hear, hear!
Here’s the thing: you might THINK you have a low-risk population, but when it all goes off the deep end, you’re still the one struggling to keep a lid on it. So yeah, I want to be in the place that sees it all enough to have it drilled into them regularly.
I tell everyone who will listen that my pregnancy with Spawn was going along just fine right up until the blood test came back that said I was in Class 1 HELLP syndrome at 26 weeks.
My husband’s cousin’s pregnancy was uncomplicated (including passing multiple GD tests) right up until her son had a shoulder dystocia that required abdominal rescue to resolve.
I think it’s hard for people to understand that not all health conditions have a gradual onset or worsening of severity. Well-meaning people ask me questions trying to suss out what obvious symptom of HELLP I missed before Spawn was born – and they don’t like to hear that the main risk factor was being pregnant.
Amen to your last paragraph! I also had HELLP Class 1 – I entered the hospital at 23 weeks 4 days and delivered very early the next morning.
I get the feeling no one really believed me when I told them there were no signs; everyone said “there must have been something!” But sometimes there isn’t. Even what I believe were early flags for preeclampsia could be put down to normal pregnancy symptoms; I will never know.
I was going to be a first time mom, no risk factors to speak of. I was fine until I wasn’t. Pregnancy does that more often than people like to think.
Yup. This was one reason I decided against a CPM-led homebirth and I was really glad of it when I had a vaguely rare complication with my second kid. My CNM, who was going on 1000 births by the time she delivered my son, told me that this complication occurred in maybe one out of 50 births. I wouldn’t have wanted to be delivering at home with some “midwife” who’d seen 100 births and had seen this complication maybe a couple times or maybe not at all.