- Patients in clinical practice are different from those enrolled in RCTs
- Drug adherence in clinical practice is likely to be lower than that achieved in RCTs, resulting in lower efficacy.
For clinical practice to provide the evidence really needed, the clinician would have to see patients and assign treatments using one of the top four approaches listed in the hierarchy of evidence below. Entries are in the order of strongest evidence requiring the least assumptions to the weakest evidence. Note that crossover studies, when feasible, even surpass randomized studies of matched identical twins in the quality and relevance of information they provide.
Let Pi denote patient i and the treatments be denoted by A and B. Thus P2B represents patient 2 on treatment B. P1 represents the average outcome over a sample of patients from which patient 1 was selected. HTE is heterogeneity of treatment effect.
|6-period crossover||P1A vs P1B (directly measure HTE)|
|2-period crossover||P1A vs P1B|
|RCT in idential twins||P1A vs P1B|
|∥ group RCT||P1A vs P2B, P1=P2 on avg|
|Observational, good artificial control||P1A vs P2B, P1=P2 hopefully on avg|
|Observational, poor artificial control||P1A vs P2B, P1≠ P2 on avg|
|Real-world physician practice||P1A vs P2B|
Regarding adherence, proponents of "real world" evidence advocate for estimating treatment effects in the context of making treatment adherence low as in clinical practice. This would result in lower efficacy and the abandonment of many treatments. It is hard to argue that a treatment should not be available for a potentially adherent patient because her fellow patients were poor adherers. Note that an RCT is the best hope for estimating efficacy as a function of adherence, through for example an instrumental variable analysis (the randomization assignment is a truly valid instrument). Much more needs to be said about how to handle treatment adherence and what should be the target adherence in an RCT, but overall it is a good thing that RCTs do not mimic clinical practice. We are entering a new era of pragmatic clinical trials. Pragmatic trials are worthy of in-depth discussion, but it is not a stretch to say that the chief advantage of pragmatic trials is not that they provide results that are more relevant to clinical practice but that they are cheaper and faster than traditional randomized trials.
Updated 2017-06-25 (last paragraph regarding adherence)