In the recent affirmative action decision by the U.S. Supreme Court, Justice Jackson made a startling claim:

Dr. Vinay Prasad takes issue with the shoddy study on which Justice Jackson might well have relied upon in good faith. I will assume that neither she nor her law clerks have the necessary expertise for critically analyzing the study she cited for making the claim that Black doctors are twice as good at saving the lives of Black newborns. In this article, Dr. Prasad shows the skepticism one needs to show upon hearing such an extraordinary claim.

The paper in question is catastrophically flawed. First, consider that it is a bold claim that a white doctor is twice as likely to kill a black baby. The effect size (TWICE as likely!) is massive. . . .

Next, in my podcast from Aug 2020 I discuss why this paper is flawed (full podcast is 91 min. but relevant discussion runs from 1:31:00 to 0:52:00 mark). Those notes are also captured here.

• If white doctors have so much worse outcomes, one would expect they are making different decisions in the care of neonates than Black doctors— but this paper cannot show the mechanism of the difference
• The paper assumes doctor-baby pairings are quasi randomized, but that is unfounded assumption. It may not be quasi randomized and well off Blacks may be more likely to have Black doctors
• A baby born is seen by a team of doctors— pediatricians, anesthesiologists, obs— which doctor is ascribed the ‘assigned provider’ per baby. What determines this assignment? (the authors do not provide this information)
• Since, the paper was published it was revealed that some hospitals put a treating doctor on the form and others put the head of the unit. (massive bias)
• A baby born is seen by a team— nurses, staff, doctors, etc— why are the races (and racial concordance) of these people not accounted for.
• If a baby gets sick, and goes to NICU and dies, which doctor is ascribed responsibility. If NICU doctors have different racial make up than other doctors could this not bias results?
• Broader issues of administrative data/ multiple hypothesis testing detailed in the episode.

Prasad also breaks down a second article claiming that Black doctors are substantially better at saving Black lives: “The Supreme Court’s affirmative action decision puts lives at risk.”

Prasad sets forth the limitations on this second study, which also makes an extraordinary race-based claim:

• Effect size is preposterously large
• No proof that the black doctors are caring for black patients (what is the mechanism?)
• Race is defined from self reports. But not all racial groups self report equally, and how do I know it doesn’t vary by geography/ region/ county?
• Why is the exposure: the ratio. Why not just # of black doctors per capita, or # of black doctors per capita black people? Did the authors preregister this ratio? Did they try other things before settling on this? How many other groups and how many other analytic plans exist? According to this ratio, if I fired 10% of the white doctors, I would improve life expectancy for blacks. Bizarre.
• The model is complicated, but my reading is that the mortality changes seen here arose between 0 and 10 years. Given the data is coded in only 3 times (and each 5 years is time 0,1,2,) we expect the ratio in 2019 to reflect the immediate impact on life expectancy, and the ratio in the 2 prior times to work over 5 and 10 years max. How do PCPs impact survival so dramatically and so quickly? And doesn’t the inclusion of 2019 ratio and outcomes imply we are also looking for immediate effects.
• Because of the choice of the ratio, the authors have to discard 50% of data b/c they could not find even 1 self reported black PCP in a county. How would the data look, if these points were included and the exposure was # black PCPs/ per capita or per capita black people? Also this to me is the biggest finding in and of itself! No need to gild the lily and offer a dubious association.
• Adjusting for poverty rate is not the same as adjusting for SES.
• Analytic flexibility is astronomical , “after testing several models for the level 1 residuals (eg, homoscedastic, autoregressive error structure, etc), mixed-effects growth models with an unstructured residual covariance matrix were used (1) to regress life-expectancy, age-adjusted all-cause mortality rates, and a log-transformed measure of mortality rate disparity between Black and White individuals on the log-transformed representativeness ratio within each county”. Analytic plans for things like this should be pre-registered.
• Note, the way PCP is coded— isn’t it likely some sub-specialists are miscoded as PCPs? How do I know this doesn’t vary by geography, location and race?
• What about counties with more vs. less cross county health care receipt? Counties adjacent to large cities, etc? Does this not vary by geography and time?
• How we do we know direction? Do more black physicians as fraction of all physicians standardized for blacks in the population improve black life expectancy? Or people (richer/ healthier) retire and move to regions that culturally suit them, and have doctors they prefer?

Prasad’s conclusions?:

[T]he claim that Black doctors lower mortality rates is entirely unsupported and unlikely to be true. Some tiny experimental data show greater adherence to a handful of interventions— but these studies may not be reproduced and moreover do not show impacts on hard outcomes. The claims of these two research papers, are entirely unfounded. If the topic was something that did not have strong sociopolitical importance, the papers would likely have been rejected repeatedly for weak methods.

What does it say about science and scientists that many appear more concerned with politics than truth? It should scare us. COVID has shown that partisanship can be so blinding. A smart doctor can say with a straight face that two year olds should cloth mask except when they nap. The power of motivated reasoning is strong.