Experimental designs are critical for obvious reasons but they have a few critical flaws, that mostly all reduce to the fact you can't randomize manipulations with everything. Whether it be due to ethics or practical constraints, you can't conduct a RCT all the time.
This can be more subtly critical than it might seem, in that even if you can manipulate some proxy, often that proxy is insufficient in actually representing the phenomenon of interest, or the conditions under which they actually occur.
I often use the example of videogames and aggression. There were plenty of experimental studies of this but it was always questionable whether lab-induced anger is the same thing as, say, the sort of violence we generally are concerned about societally.
I generally have tried to teach students that experimental designs when done right provide powerful causal evidence of something, but often with limited generalizability; observational designs in contrast provide powerful generalizable evidence of some kind of association, but often with limited certainty about the causal pathways involved.
I've been in a department that was rabidly experimental in its focus and it always seemed sort of short-sighted, because people were idolizing RCTs with proxy manipulations that had questionable generalizability to the real-world phenomena they were trying to model.
Ideally you'd bring both experimental and observational evidence to bear on a question. Your conclusions should be robust to different types of designs.
Yes. Even conceding every point made in this article, RCTs are still the gold standard. The article points out that RCTs are often impractical, expensive or ethically challenging. Well, so is the gold standard of anything.
OP suggests that alternative methods like target trial emulation, propensity scoring and double machine learning can be used to approximate the conditions of an RCT using existing data. In saying so he gives away the tell, which is that RCTs are the standard being aspired to.
Observational data may well be undervalued. But RCTs are still the gold standard.
"Advocates of evidence based medicine have criticised the adoption of interventions evaluated by using only observational data. We think that everyone might benefit if the most radical protagonists of evidence based medicine organised and participated in a double blind, randomised, placebo controlled, crossover trial of the parachute."
That's not a "classic". It's just a silly, overwrought straw-man [1] that gets passed around by people who think they're being clever, and/or don't understand the topic with any degree of nuance.
Nobody reasonable suggests that RCTs must be used for every problem. When the effect size is enormous, or there's obviously no alternative (as in the parachute example), it's silly to suggest the use of an experiment.
On the other hand, if you want to compare some subtle difference between two competing parachute designs, for example, then perhaps an RCT would actually be appropriate. This is much closer to what is actually being measured in real clinical trials.
[1] To be completely fair to the authors, it was intended as a joke, not an argument. The real joke ended up being that people took it seriously.
Good article. I will only add that I think the problems outlined in the article are exacerbated by the regulatory apparatus. If this were a debate related to truth seeking alone, it would be good. But trials drive banning/unbanning of treatments and medicines, as well as their mandated coverage by insurance.
We could be much more flexible in our approach to things if we would un-ban things. For example after phase 1 trial or based on sufficient observational evidence, things can no longer be banned but have a higher standard for "insurance is now required to cover it".
Quite thought-provoking, and connecting it to a related field it seems the (relative) success of LLMs and the likes are indications that enough data can at least learn you something without always needing to interfere with the world first(?)
Let me restate that: Enough accurate data about the real world can let you learn something about the real world without having to yourself interface with the real world. So, for example, Kepler's Laws were based on Tycho Brahe's observations, not on Kepler's.
In medicine, observational evidence is actually better and far more ethical than the RCT. (Which simply dooms the terminally ill to fake treatment.) You just need large datasets and an agile culture that's responsive to new input.
Don't forget that RCTs are very far from perfect and issues -- sometimes literally fatal issues -- have later turned up via observational evidence in large cohorts. Vioxx, for instance. Many others.
I believe, without the tiniest shred of doubt, that the only trials drugs need to go through are initial safety/toxicity trials (phases 0/1) and that everything else would be much better left to access+observation.
This feels off. In medicine, any evidence can also be blinded by confounding factors that are far easier to miss without adding specific controls. Really, in any field this will be the case.
Should we demand an RCT before we accept evidence? Of course not. At some point you do have to make a choice on things.
And it should be noted that most drugs do have early cutoff criteria if the evidence is strong enough that it is working. It isn't like people are wanting to withhold good treatments from the world. Adding controls and randomizing them, though, has proven to be highly effective at helping progress.
> "This feels off. In medicine, any evidence can also be blinded by confounding factors that are far easier to miss without adding specific controls. Really, in any field this will be the case."
If you have enough data, you can smooth out individual fluctuations due to things like drug interactions, non-compliance, etc. (And indeed you might discover drug interactions!) Observational trials ultimately mirror how drugs are used in the real world.
> "Adding controls and randomizing them, though, has proven to be highly effective at helping progress."
I would argue just the opposite. Demands for increasingly byzantine trials have ballooned the costs associated with drug development, and have slowed things to a crawl. There's a reason the field's golden age was in the 1940s and 1950s, and it's not just "low hanging fruit." Today nobody in their right mind wants to work in drug development when they could work in tech or even finance.
You are simultaneously arguing for a more complex and nuanced testing approach that demands much higher quantities of data as a result, and also against RCTs, which perhaps rightly you've identified as having suffered from the same kind of cost disease as all other health care in the USA. I can't help but feel like you've identified the wrong root cause here.
Right, but you are just relying on a different form of random, there. The whole point of making controls and then building experiments on changing them, is to get more power from fewer observations. No?
Again, it is off to think that one is automatically superior to the other. Certainly to the exclusion of the other. And that is what feels off with the framing of the parent post. I am perfectly fine saying you should use both observational and controlled trials. But I think it is also wrong to think you don't have to build experiments to test interventions.
This is why you put metrics in your service code. So that you can observe them behave and look for things to change. This is also why you do test cases on your code, so that you can specifically target your change.
Now, I fully back the idea that just A/B testing something doesn't automatically mean you learn something true. But neither does observing a strong outcome on uncontrolled data.
"Large controlled experiments are costly and can hurt people who opt-in to informed consent. Instead, we should do significantly, significantly larger experiments, with undefined success/failure conditions, and no informed consent."
How do you get enough data? If, for example, you need a lot of people in the sample, that might not be so easy. In the abstract, should it not come done to what is the best experimental design for each case?
> Demands for increasingly byzantine trials have ballooned the costs associated with drug development,
Even if that is true, is it an intrinsic problem with trials or just bad regulation? If it is the latter then you need to change the regulations? Is the problem global - is every regulator everywhere demanding byzantine trails?
Regulators don't demand byzantine trials. They'd prefer simpler ones for a million different reasons. Byzantine trial designs exist because it's incredibly hard to prove your drug works even in an RCT. But GP thinks you can just look at EMR data and ta-da now you know, lol.
>Operation Sea-Spray was a 1950 U.S. Navy secret biological warfare experiment in which Serratia marcescens and Bacillus globigii bacteria were sprayed over the San Francisco Bay Area in California in order to determine how vulnerable a city like San Francisco would be to a bioweapon attack. There has been speculation that the experiment may have contributed to one death and at least ten illnesses.
Also, despite GP's dismissal of the "low-hanging fruit" hypothesis, it is obviously true that we've found the easy-to-discover drugs, and therefore drugs would get harder and harder to find.
A flippant dismissal in an HN comment does not actually negate reality.
FDA has essentially one requirement: prove that your drug is safe and effective.
The reason trial designs get more and more byzantine is because the drugs themselves work less-and-less well. They're far more nuanced and precise. The experiments have to be extremely well-controlled, and then this has to balance against cost/timeline of the trial, and that's why sponsors choose to use byzantine trial designs.
Proving "efficacy" -- which is the difficult and expensive part -- should not be necessary, and the government increasingly moves its own goalposts as to what the word efficacy even means. Simple as that. Postmarketing surveillance can easily determine what's effective and what's not, and medical orgs can adjust.
Brilliant idea. This way both actual working drugs and vitamin-aisle potions look identical to consumers. Neither (or both?) can make claims to their effectiveness since they're both backed by the same (lack of) actual knowledge.
This is especially great because it puts anyone who actually wants to actually make a working drug at a significant disadvantage. It'll take them longer to get to market, cost them a billion dollars more, then their medicine gets to sit next to a thousand variations of "Vitamin C for Leukemia" that all cost a lot less.
There would be virtually no incentive for anyone to make an actual drug.
> Postmarketing surveillance can easily determine what's effective and what's not, and medical orgs can adjust.
"Easily" is doing a ton of work. Postmarketing surveillance can sometimes give low-confidence signal as to what's effective and what's not.
Sure, in the same way "the market" discovered that cigarettes cause significant health problems.
Step 1. Ensure extremely widespread use to ensure that effects are noticed relatively quickly (i.e. within a few decades)
Step 2. Hope someone notices correlations between that drug use and the positive/negative downstream effects.
Step 3. Hope someone who has access to the data required to actually conduct an observational study actually wants to run the observational study. By the way, you better hope all the info you need is available in insurance claims data (it's not) because that's really the only data you'll have at high scale. For most of these fraudulent drugs, you won't even have that! Because why would you go through the hassle of gaining reimbursement when you can just sell bullshit for cash? So in practice the required real-world data on these drugs and their effects literally will not exist in most cases. Where the data does exist, it'll be confounded by the unrecorded use of all the other fake drugs too.
Step 4. Hope the company selling a non-working drug doesn't have the clever idea of publishing countervailing "studies" backed by much larger budgets and with more existential motivation. They won't be assessed by FDA so these studies can just say whatever they want, of course.
Step 5. Lobby for outlawing of a drug that, in fact, didn't break any law whatsoever. Obviously it can't be illegal to sell a non-working in a regime that has no way to assess whether a drug works. So you really just gotta hope to post enough times that a drug is bad that people stop taking it.
But cigarettes' negative effects are significantly stronger than most drugs' positive (or negative) effects. So every single step is going to be dramatically harder, require more people taking the drug, more data, more time, more sophisticated analysis, more direct scrutiny from "the market."
Give it a few decades and, maybe a couple hundred or couple thousand deaths, and with a bit of luck "the market" will have eliminated one of the thousands of fake drugs proliferating across the market.
Thankfully for the company behind it, they can always just start up another totally legal fraud by just picking a different chemical (inert or not) and claiming a different medical benefit.
Oh yeah, meanwhile, doctors are just flooded with absolute bullshit claims 24/7 with no repercussions for anyone just lying to them. It's good stuff.
Literally one of the dumbest ideas I've ever heard.
> But cigarettes' negative effects are significantly stronger than most drugs' positive (or negative) effects
I am clearly talking about both positive and negative effects of drugs. Both would be generally extremely hard to detect, and literally impossible to detect without both 1) massive consumption of that drug and 2) a civilizational-scale rewriting of the data infrastructure that actually exists.
Perhaps you should explore Capitalism 201. You'd learn that markets only have all their most lovely features in an environment of complete information, which this is not. The more the information asymmetry (or complete ignorance) there is, the less-well markets work.
The proposal being put forth here is: what if we just had virtually no actual information about these products?
>I am clearly talking about both positive and negative effects of drugs.
Then you are answering to a point that was not raised.
> most lovely features in an environment of complete information, which this is not. The more the information asymmetry (or complete ignorance) there is, the less-well markets work...
I am well aware. But for what I as well as the parent is saying, this only require information regarding if the thing works or not for the people who use it as long as that information is not censored.
Uhh... the question was "can the market detect efficacy." My answer is: Generally no, and it also can't detect negative effects either.
That's answering the question.
> this only require information regarding if the thing works or not for the people who use it as long as that information is not censored.
Can "the market" currently detect whether there is complex life in the Messier 54 star cluster?
No.
Is it because that information is censored?
No.
It's because that information doesn't exist.
Or more specifically, because that information is indistinguishable from noise. Even in the scenario of cigarettes' extremely strong effect (stronger than nearly every drug) deployed at massive scale (higher scale than nearly every drug), it required decades to establish causality.
Especially in the regime of drugs being released to market solely on safety data, the information about whether they work would be generally indistinguishable from noise. For way, way longer than cigarettes given the smaller population, smaller effect sizes, and the (presumed) proliferation of noisy concomitant "medications."
Do you think companies spend billions of dollars running RCTs for fun? The FDA doesn't require RCTs, they require only strong evidence of efficacy and safety. They do ultra-expensive RCTs because without them, it's virtually impossible for anyone to know – including the patients themselves – whether the drugs work.
>because that information is indistinguishable from noise.
Not if the signal is strong. Whether a thing works or not, is a pretty strong signal. Your equating it to bad effects of smoking is flawed and is not comparable.
You other arugments are also pretty bizzare, but not going into that for sake of focus.
> your specific example of it taking decades of massive deployment and close scrutiny to detect an effect significantly stronger than most drugs is ummm... flawed and not comparable for no reason I can articulate
okay buddy
That's okay, you're someone who thinks the FDA is in cahoots with insurers and drug companies to approve ineffective or unsafe drugs at scale lmao. And here you are arguing that the market would actually naturally detect exactly that.
That is correct. Smoking has a stronger effect on cancer rates than paracetamol does on fever (which itself is an extremely strong effect that can be proven in RCTs of just a few dozen people - far stronger than most drugs on the market).
> In medicine, observational evidence is actually better and far more ethical than the RCT. (Which simply dooms the terminally ill to fake treatment.)
This is just nonsense. First, everyone in a trial is informed of the situation. It's not "unethical" unless you lie about it. If you participate in a trial, you do so knowing that you might not get the experimental drug. It's a selfless, honorable thing to do, and we shouldn't be framing it as some kind of scam.
Second, we don't give terminally ill people "fake treatment" (placebo trials). We give them current standard of care. Giving someone a placebo trial doesn't prove anything that would change clinical practice, because you want to know if the drug works better than what is out there today. Rarely is that standard of care "nothing", and this (bad controls) is actually a primary reason that a lot of drug company trials are rejected by the FDA.
If I didn't see the Wall Street Journal editorial board repeating the same garbage in defense of patent medicines, I'd write you off as simply having a sophomoric understanding of how trials work. I'm convinced that someone is driving this absurd narrative.
> Which simply dooms the terminally ill to fake treatment.
I wish people would stop saying this. First, controls aren't necessarily "fake treatment", they are often compared to other standard treatments.
Second, the treatment being tested can actually harm the patient more, therefore the people receiving your alleged "fake treatment" can actually come out better off. Which is the "fake treatment" now?
I don't disagree with your final point, but mainly with this increasingly pervasive and wrong framing of RCTs.
So if both the standard of care as well as the drug being trials have the same side effects, would they be included in the package insert/drug literature? Because the incidence rate of it would be same for both the groups?
I think it’s pointless to quibble about very specific things like this, but that gets thrown out the window when someone claims “Definitions are the very basics in writing, check them before you publish” and then use their own nonstandard definition!
Seems like they wanted to nitpick about term usage when:
a) the usage they’re taking issue with is actually fairly commonplace (e.g. https://www.who.int/news/item/27-10-2017-who-to-review-its-i...)
b) their own suggested replacement is very heterodox (I’ve certainly never heard anyone call it that).
Mentioned, yes, but not correctly. They should have put a "(RCT)" immediately after the first instance of "randomized control trial" at the very least. However, I'm of the opinion that the acronym should have been expanded it the first paragraph, or even first sentence, since it's used in the subheader. Either way, this was definitely a failure on the author's part.
The reality that individuals and cultures create can be a dark and confining place. Empiricism is our only window to a universe full of possibility and light. Humanity is like a child, standing on our toes to peer through and wonder.
Experimental designs are critical for obvious reasons but they have a few critical flaws, that mostly all reduce to the fact you can't randomize manipulations with everything. Whether it be due to ethics or practical constraints, you can't conduct a RCT all the time.
This can be more subtly critical than it might seem, in that even if you can manipulate some proxy, often that proxy is insufficient in actually representing the phenomenon of interest, or the conditions under which they actually occur.
I often use the example of videogames and aggression. There were plenty of experimental studies of this but it was always questionable whether lab-induced anger is the same thing as, say, the sort of violence we generally are concerned about societally.
I generally have tried to teach students that experimental designs when done right provide powerful causal evidence of something, but often with limited generalizability; observational designs in contrast provide powerful generalizable evidence of some kind of association, but often with limited certainty about the causal pathways involved.
I've been in a department that was rabidly experimental in its focus and it always seemed sort of short-sighted, because people were idolizing RCTs with proxy manipulations that had questionable generalizability to the real-world phenomena they were trying to model.
Ideally you'd bring both experimental and observational evidence to bear on a question. Your conclusions should be robust to different types of designs.
Yes. Even conceding every point made in this article, RCTs are still the gold standard. The article points out that RCTs are often impractical, expensive or ethically challenging. Well, so is the gold standard of anything.
OP suggests that alternative methods like target trial emulation, propensity scoring and double machine learning can be used to approximate the conditions of an RCT using existing data. In saying so he gives away the tell, which is that RCTs are the standard being aspired to.
Observational data may well be undervalued. But RCTs are still the gold standard.
One classic paper for on this topic:
_Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials_
https://pmc.ncbi.nlm.nih.gov/articles/PMC300808/
"Advocates of evidence based medicine have criticised the adoption of interventions evaluated by using only observational data. We think that everyone might benefit if the most radical protagonists of evidence based medicine organised and participated in a double blind, randomised, placebo controlled, crossover trial of the parachute."
That's not a "classic". It's just a silly, overwrought straw-man [1] that gets passed around by people who think they're being clever, and/or don't understand the topic with any degree of nuance.
Nobody reasonable suggests that RCTs must be used for every problem. When the effect size is enormous, or there's obviously no alternative (as in the parachute example), it's silly to suggest the use of an experiment.
On the other hand, if you want to compare some subtle difference between two competing parachute designs, for example, then perhaps an RCT would actually be appropriate. This is much closer to what is actually being measured in real clinical trials.
[1] To be completely fair to the authors, it was intended as a joke, not an argument. The real joke ended up being that people took it seriously.
Good article. I will only add that I think the problems outlined in the article are exacerbated by the regulatory apparatus. If this were a debate related to truth seeking alone, it would be good. But trials drive banning/unbanning of treatments and medicines, as well as their mandated coverage by insurance.
We could be much more flexible in our approach to things if we would un-ban things. For example after phase 1 trial or based on sufficient observational evidence, things can no longer be banned but have a higher standard for "insurance is now required to cover it".
Damn fine article, lovely conclusion, a real pleasure to read
Quite thought-provoking, and connecting it to a related field it seems the (relative) success of LLMs and the likes are indications that enough data can at least learn you something without always needing to interfere with the world first(?)
Let me restate that: Enough accurate data about the real world can let you learn something about the real world without having to yourself interface with the real world. So, for example, Kepler's Laws were based on Tycho Brahe's observations, not on Kepler's.
In medicine, observational evidence is actually better and far more ethical than the RCT. (Which simply dooms the terminally ill to fake treatment.) You just need large datasets and an agile culture that's responsive to new input.
Don't forget that RCTs are very far from perfect and issues -- sometimes literally fatal issues -- have later turned up via observational evidence in large cohorts. Vioxx, for instance. Many others.
I believe, without the tiniest shred of doubt, that the only trials drugs need to go through are initial safety/toxicity trials (phases 0/1) and that everything else would be much better left to access+observation.
This feels off. In medicine, any evidence can also be blinded by confounding factors that are far easier to miss without adding specific controls. Really, in any field this will be the case.
Should we demand an RCT before we accept evidence? Of course not. At some point you do have to make a choice on things.
And it should be noted that most drugs do have early cutoff criteria if the evidence is strong enough that it is working. It isn't like people are wanting to withhold good treatments from the world. Adding controls and randomizing them, though, has proven to be highly effective at helping progress.
> "This feels off. In medicine, any evidence can also be blinded by confounding factors that are far easier to miss without adding specific controls. Really, in any field this will be the case."
If you have enough data, you can smooth out individual fluctuations due to things like drug interactions, non-compliance, etc. (And indeed you might discover drug interactions!) Observational trials ultimately mirror how drugs are used in the real world.
> "Adding controls and randomizing them, though, has proven to be highly effective at helping progress."
I would argue just the opposite. Demands for increasingly byzantine trials have ballooned the costs associated with drug development, and have slowed things to a crawl. There's a reason the field's golden age was in the 1940s and 1950s, and it's not just "low hanging fruit." Today nobody in their right mind wants to work in drug development when they could work in tech or even finance.
You are simultaneously arguing for a more complex and nuanced testing approach that demands much higher quantities of data as a result, and also against RCTs, which perhaps rightly you've identified as having suffered from the same kind of cost disease as all other health care in the USA. I can't help but feel like you've identified the wrong root cause here.
Right, but you are just relying on a different form of random, there. The whole point of making controls and then building experiments on changing them, is to get more power from fewer observations. No?
Again, it is off to think that one is automatically superior to the other. Certainly to the exclusion of the other. And that is what feels off with the framing of the parent post. I am perfectly fine saying you should use both observational and controlled trials. But I think it is also wrong to think you don't have to build experiments to test interventions.
This is why you put metrics in your service code. So that you can observe them behave and look for things to change. This is also why you do test cases on your code, so that you can specifically target your change.
Now, I fully back the idea that just A/B testing something doesn't automatically mean you learn something true. But neither does observing a strong outcome on uncontrolled data.
Yeah, GP is basically saying:
"Large controlled experiments are costly and can hurt people who opt-in to informed consent. Instead, we should do significantly, significantly larger experiments, with undefined success/failure conditions, and no informed consent."
Insane opinion
How do you get enough data? If, for example, you need a lot of people in the sample, that might not be so easy. In the abstract, should it not come done to what is the best experimental design for each case?
> Demands for increasingly byzantine trials have ballooned the costs associated with drug development,
Even if that is true, is it an intrinsic problem with trials or just bad regulation? If it is the latter then you need to change the regulations? Is the problem global - is every regulator everywhere demanding byzantine trails?
Regulators don't demand byzantine trials. They'd prefer simpler ones for a million different reasons. Byzantine trial designs exist because it's incredibly hard to prove your drug works even in an RCT. But GP thinks you can just look at EMR data and ta-da now you know, lol.
> There's a reason the field's golden age was in the 1940s and 1950s
Yes, that was because of things like
https://en.wikipedia.org/wiki/Tuskegee_syphilis_experiment
https://en.wikipedia.org/wiki/Stateville_Penitentiary_Malari...
https://en.wikipedia.org/wiki/Operation_Sea-Spray
https://en.wikipedia.org/wiki/Nazi_human_experimentation
I understand that certain people are salivating at the thought of a return to those times; I'm not one of them.
>Operation Sea-Spray was a 1950 U.S. Navy secret biological warfare experiment in which Serratia marcescens and Bacillus globigii bacteria were sprayed over the San Francisco Bay Area in California in order to determine how vulnerable a city like San Francisco would be to a bioweapon attack. There has been speculation that the experiment may have contributed to one death and at least ten illnesses.
Wow.
Also, despite GP's dismissal of the "low-hanging fruit" hypothesis, it is obviously true that we've found the easy-to-discover drugs, and therefore drugs would get harder and harder to find.
A flippant dismissal in an HN comment does not actually negate reality.
> Demands for increasingly byzantine trials
This is silly.
FDA has essentially one requirement: prove that your drug is safe and effective.
The reason trial designs get more and more byzantine is because the drugs themselves work less-and-less well. They're far more nuanced and precise. The experiments have to be extremely well-controlled, and then this has to balance against cost/timeline of the trial, and that's why sponsors choose to use byzantine trial designs.
Proving "efficacy" -- which is the difficult and expensive part -- should not be necessary, and the government increasingly moves its own goalposts as to what the word efficacy even means. Simple as that. Postmarketing surveillance can easily determine what's effective and what's not, and medical orgs can adjust.
Brilliant idea. This way both actual working drugs and vitamin-aisle potions look identical to consumers. Neither (or both?) can make claims to their effectiveness since they're both backed by the same (lack of) actual knowledge.
This is especially great because it puts anyone who actually wants to actually make a working drug at a significant disadvantage. It'll take them longer to get to market, cost them a billion dollars more, then their medicine gets to sit next to a thousand variations of "Vitamin C for Leukemia" that all cost a lot less.
There would be virtually no incentive for anyone to make an actual drug.
> Postmarketing surveillance can easily determine what's effective and what's not, and medical orgs can adjust.
"Easily" is doing a ton of work. Postmarketing surveillance can sometimes give low-confidence signal as to what's effective and what's not.
>There would be virtually no incentive for anyone to make an actual drug.
Dumb question: Wouldn't the market discover what works and what does not automatically, like any other product?
Sure, in the same way "the market" discovered that cigarettes cause significant health problems.
Step 1. Ensure extremely widespread use to ensure that effects are noticed relatively quickly (i.e. within a few decades)
Step 2. Hope someone notices correlations between that drug use and the positive/negative downstream effects.
Step 3. Hope someone who has access to the data required to actually conduct an observational study actually wants to run the observational study. By the way, you better hope all the info you need is available in insurance claims data (it's not) because that's really the only data you'll have at high scale. For most of these fraudulent drugs, you won't even have that! Because why would you go through the hassle of gaining reimbursement when you can just sell bullshit for cash? So in practice the required real-world data on these drugs and their effects literally will not exist in most cases. Where the data does exist, it'll be confounded by the unrecorded use of all the other fake drugs too.
Step 4. Hope the company selling a non-working drug doesn't have the clever idea of publishing countervailing "studies" backed by much larger budgets and with more existential motivation. They won't be assessed by FDA so these studies can just say whatever they want, of course.
Step 5. Lobby for outlawing of a drug that, in fact, didn't break any law whatsoever. Obviously it can't be illegal to sell a non-working in a regime that has no way to assess whether a drug works. So you really just gotta hope to post enough times that a drug is bad that people stop taking it.
But cigarettes' negative effects are significantly stronger than most drugs' positive (or negative) effects. So every single step is going to be dramatically harder, require more people taking the drug, more data, more time, more sophisticated analysis, more direct scrutiny from "the market."
Give it a few decades and, maybe a couple hundred or couple thousand deaths, and with a bit of luck "the market" will have eliminated one of the thousands of fake drugs proliferating across the market.
Thankfully for the company behind it, they can always just start up another totally legal fraud by just picking a different chemical (inert or not) and claiming a different medical benefit.
Oh yeah, meanwhile, doctors are just flooded with absolute bullshit claims 24/7 with no repercussions for anyone just lying to them. It's good stuff.
Literally one of the dumbest ideas I've ever heard.
Not sure, I was just talking about the efficacy, not long term side effects. The parent comment was also talking about showing efficacy...
>Literally one of the dumbest ideas I've ever heard.
It is capitalism 101. Not sure if that is the dumbest idea, because the whole world seem to run on it.
> But cigarettes' negative effects are significantly stronger than most drugs' positive (or negative) effects
I am clearly talking about both positive and negative effects of drugs. Both would be generally extremely hard to detect, and literally impossible to detect without both 1) massive consumption of that drug and 2) a civilizational-scale rewriting of the data infrastructure that actually exists.
Perhaps you should explore Capitalism 201. You'd learn that markets only have all their most lovely features in an environment of complete information, which this is not. The more the information asymmetry (or complete ignorance) there is, the less-well markets work.
The proposal being put forth here is: what if we just had virtually no actual information about these products?
Well, the markets don't work. Capitalism 201.
>I am clearly talking about both positive and negative effects of drugs.
Then you are answering to a point that was not raised.
> most lovely features in an environment of complete information, which this is not. The more the information asymmetry (or complete ignorance) there is, the less-well markets work...
I am well aware. But for what I as well as the parent is saying, this only require information regarding if the thing works or not for the people who use it as long as that information is not censored.
Uhh... the question was "can the market detect efficacy." My answer is: Generally no, and it also can't detect negative effects either.
That's answering the question.
> this only require information regarding if the thing works or not for the people who use it as long as that information is not censored.
Can "the market" currently detect whether there is complex life in the Messier 54 star cluster?
No.
Is it because that information is censored?
No.
It's because that information doesn't exist.
Or more specifically, because that information is indistinguishable from noise. Even in the scenario of cigarettes' extremely strong effect (stronger than nearly every drug) deployed at massive scale (higher scale than nearly every drug), it required decades to establish causality.
Especially in the regime of drugs being released to market solely on safety data, the information about whether they work would be generally indistinguishable from noise. For way, way longer than cigarettes given the smaller population, smaller effect sizes, and the (presumed) proliferation of noisy concomitant "medications."
Do you think companies spend billions of dollars running RCTs for fun? The FDA doesn't require RCTs, they require only strong evidence of efficacy and safety. They do ultra-expensive RCTs because without them, it's virtually impossible for anyone to know – including the patients themselves – whether the drugs work.
>because that information is indistinguishable from noise.
Not if the signal is strong. Whether a thing works or not, is a pretty strong signal. Your equating it to bad effects of smoking is flawed and is not comparable.
You other arugments are also pretty bizzare, but not going into that for sake of focus.
Well, with an argument like that!
> if the signal is strong, it will be visible
> your specific example of it taking decades of massive deployment and close scrutiny to detect an effect significantly stronger than most drugs is ummm... flawed and not comparable for no reason I can articulate
okay buddy
That's okay, you're someone who thinks the FDA is in cahoots with insurers and drug companies to approve ineffective or unsafe drugs at scale lmao. And here you are arguing that the market would actually naturally detect exactly that.
> to detect an effect significantly stronger than most drugs..
What the heck are you smokin? Did paracetamol have a weaker affect on temperature than smoking causing cancer?
That is correct. Smoking has a stronger effect on cancer rates than paracetamol does on fever (which itself is an extremely strong effect that can be proven in RCTs of just a few dozen people - far stronger than most drugs on the market).
Thank you for demonstrating my point.
And thank you for demonstrating mine..
> In medicine, observational evidence is actually better and far more ethical than the RCT. (Which simply dooms the terminally ill to fake treatment.)
This is just nonsense. First, everyone in a trial is informed of the situation. It's not "unethical" unless you lie about it. If you participate in a trial, you do so knowing that you might not get the experimental drug. It's a selfless, honorable thing to do, and we shouldn't be framing it as some kind of scam.
Second, we don't give terminally ill people "fake treatment" (placebo trials). We give them current standard of care. Giving someone a placebo trial doesn't prove anything that would change clinical practice, because you want to know if the drug works better than what is out there today. Rarely is that standard of care "nothing", and this (bad controls) is actually a primary reason that a lot of drug company trials are rejected by the FDA.
If I didn't see the Wall Street Journal editorial board repeating the same garbage in defense of patent medicines, I'd write you off as simply having a sophomoric understanding of how trials work. I'm convinced that someone is driving this absurd narrative.
[flagged]
> Which simply dooms the terminally ill to fake treatment.
I wish people would stop saying this. First, controls aren't necessarily "fake treatment", they are often compared to other standard treatments.
Second, the treatment being tested can actually harm the patient more, therefore the people receiving your alleged "fake treatment" can actually come out better off. Which is the "fake treatment" now?
I don't disagree with your final point, but mainly with this increasingly pervasive and wrong framing of RCTs.
Yes. Potentially life-saving drugs in RCTs are usually compared to "standard of care," i.e. already-approved treatments, not to placebos.
So if both the standard of care as well as the drug being trials have the same side effects, would they be included in the package insert/drug literature? Because the incidence rate of it would be same for both the groups?
Seth Roberts was arguing this ~20 years ago and would have loved the advent of LLMs...
RCT is Random Controlled Trial. Definitions are the very basics in writing, check them before you publish.
Where is it defined as "Random Controlled Trial"?
Ignoring the typo above, it's mentioned in the text: https://en.wikipedia.org/wiki/Randomized_controlled_trial
That’s not “Random Controlled Trial”.
I think it’s pointless to quibble about very specific things like this, but that gets thrown out the window when someone claims “Definitions are the very basics in writing, check them before you publish” and then use their own nonstandard definition!
Seems like they wanted to nitpick about term usage when: a) the usage they’re taking issue with is actually fairly commonplace (e.g. https://www.who.int/news/item/27-10-2017-who-to-review-its-i...) b) their own suggested replacement is very heterodox (I’ve certainly never heard anyone call it that).
Mentioned, yes, but not correctly. They should have put a "(RCT)" immediately after the first instance of "randomized control trial" at the very least. However, I'm of the opinion that the acronym should have been expanded it the first paragraph, or even first sentence, since it's used in the subheader. Either way, this was definitely a failure on the author's part.
In the author's mind
You’re the one that used the term, not the author.
Woosh
> The increasing availability of large datasets should make this an especially good time to reconsider observational evidence in many fields.
This is not happening.
> I was surprised to find that they usually discard papers based on observational evidence wholesale.
He he...welcome to the real world!
They reference RCT without first defining it.
https://en.wikipedia.org/wiki/Randomized_controlled_trial
I noticed that as well. My eyes feverishly scanning the previous paragraph for the definition.
The reality that individuals and cultures create can be a dark and confining place. Empiricism is our only window to a universe full of possibility and light. Humanity is like a child, standing on our toes to peer through and wonder.