Alexander Trevelyan writes:
I used to be blissful to see you are taking a second to point out the issues with the chilly water research that was making the rounds not too long ago. I write often about what I think about to be quite a lot of suspect practices in medical trial reporting, typically coping with misleading statistical strategies/reporting. I’m a physicist and never a statistician myself—I used to be in a gaggle that had joint conferences with Raghu Parthasarathy’s lab at Oregon—however I’ve been making an attempt to hone my understanding of medical trial stats not too long ago.
Final week, the Alzheimer’s drug Leqembi (lecanemab) was permitted by the FDA, which total appears effective, nevertheless it rekindled some debate concerning the characterization of the drug inflicting a “27% slowing in cognitive decline” over placebo; see here. This 27% determine was touted by, for instance, the NIH NIA in an announcement concerning the drug’s promise.
So right here’s my challenge, which I’d love to listen to your ideas on (since this drug is a reasonably large deal in Alzheimer’s and has been fairly controversial)—the 27% quantity is an easy share distinction that was calculated by first discovering the change in baseline for the placebo and remedy teams on the CDR-SB check (see first panel of Determine 2 within the NEJM article), then utilizing the ultimate knowledge level for every group to calculate the relative change between placebo and remedy. Does this appear as loopy to you because it does to me?
First, absolutely the distinction within the goal metric was underneath 3%. Second, calculating a share distinction on a amount that we’ve rescaled to start out at zero appears a bit… odd? It got here to my consideration as a result of a smaller outfit—one at present underneath investigation by about each three-letter federal company you’ll be able to identify—simply launched their most up-to-date medical trial outcomes, which had very small N and no error bars, however a subgroup that they touted hovered round zero they usually claimed a “200% distinction!” between the placebo and remedy teams (the uncooked knowledge factors had been a +0.6 and -0.6 change).
OK, I’ll click on by means of and have a look . . .
My first response is that it’s laborious to learn a scholarly article from an unfamiliar area! A number of subject-matter ideas that I’m not acquainted with, additionally the format is totally different from issues I normally learn, so it’s laborious for me to skim by means of to get to the important thing factors.
However, OK, this isn’t so laborious to learn, truly. I’m right here within the Strategies and Outcomes part of the summary: That they had 1800 Alzheimer’s sufferers, half obtained remedy and half obtained placebo, and their consequence is the change in rating in “Medical Dementia Score–Sum of Bins (CDR-SB; vary, 0 to 18, with larger scores indicating better impairment).” I hope they regulate for the pre-test rating; in any other case they’re throwing away data, however on this case the pattern dimension is so massive that this ought to be no large deal, we should always get approximate steadiness between the 2 teams.
In any case, right here’s the consequence: “The imply CDR-SB rating at baseline was roughly 3.2 in each teams. The adjusted least-squares imply change from baseline at 18 months was 1.21 with lecanemab and 1.66 with placebo.” So each teams obtained worse. That’s unhappy however I suppose anticipated. And I suppose that is how they obtained the 27% slowing factor: Common decline in management group was 1.66, common decline in remedy group is 1.21, you are taking 1 – 1.21/1.66 = 0.27, so a 27% slowing in cognitive decline.
Now transferring to the statistical evaluation part of the primary paper: A number of horrible stuff with significance testing and alpha values, however I can ignore all this. The sample within the knowledge appears clear. Determine 2 exhibits time tendencies for averages. I’d additionally prefer to see trajectories for people. Total, although, saying “a median 27% slowing in cognitive decline” appears cheap sufficient, given the info they present within the paper.
I despatched the above to Trevelyan, who responded:
Attention-grabbing, however now I’m nervous that possibly I spend an excessive amount of time on the background and never sufficient time in making my foremost concern extra clear. I don’t have any points with the calculation of the % distinction, per se, however slightly what it’s meant to symbolize (i.e., the remedy impact). As you famous, and is sadly the state of the sphere, the curves all the time go down in Alzheimer’s remedy—however that doesn’t must be the case! The holy grail is one thing that makes the remedy curve go up! The principle factor that set off alarm bells for me is that the “different firm” I referenced claims to have noticed an enchancment with their drug and an related 200%(!) slowing in cognitive decline. Of their case, the placebo obtained 0.6 factors worse and the remedy 0.6 factors higher, so 200%! However their remedy may’ve gotten 10 factors higher and the placebo 10 factors worse, and that’s additionally 200%! Or possibly 0.000001 factors higher versus 0.000001 factors worse—once more, 200%.
I feel my total concern is, “why are we utilizing a metric that may break in such an apparent manner underneath completely cheap (if at present aspiration) remedy outcomes?”
See here for knowledge from “different firm” if you’re curious (scroll all the way down to gentle subgroup, ugh).
And right here’s a graph made by Matthew Schrag, who’s an Alzheimer’s researcher and knowledge sleuth, which rescales the change within the metric and exhibits absolutely the change within the CDR-SB check. The inside plot exhibits the graph from the unique paper; the bigger plot is rescaled:
My reply: I’m unsure. I get your basic level, however if in case you have a 0-18 rating and it will increase from 3.2 to 4.8, that looks like a significant change, no? They’re not saying they stopped the cognitive decline, simply that they slowed it by 27%.
P.S. I talked with somebody who works on this space who says that nearly everybody in the neighborhood is skeptical concerning the claimed massive advantages for lecanemab, and in addition that there’s basic concern that sources spent on this might be higher utilized in direct providers. This isn’t to say the skeptics are essentially proper—I do know nothing about all this!—however simply to level out that there’s loads of present controversy right here.