Statistical Thinking

Posts

Minimal-Assumption Estimation of Survival Probability vs. a Continuous Variable

computing
prediction
r
regression
survival-analysis
validation
2025

There is no straightforward nonparametric smoother for estimating a smooth relationship between a continuous variable and the probability of survival past a fixed time when censoring is present. Several flexible methods are compared with regard to estimation error, and recommendations are made on the basis of a simulation study for one data generating mechanism. The results are particularly applicable to estimation of smooth calibration curves with right-censored data.

Apr 19, 2025
Frank Harrell
10 min

Statistical Computing Approaches to Maximum Likelihood Estimation

computing
data-science
inference
likelihood
ordinal
prediction
r
regression
2024

Maximum likelihood estimation (MLE) is central to estimation and development of predictive models. Outside of linear models and simple estimators, MLE requires trial-and-error iterative algorithms to find the set of parameter values that maximizes the likelihood, i.e., makes the observed data most likely to have been observed under the statistical model. There are many iterative optimization algorithms and R programming paradigms to choose from. There are also many pre-processing steps to consider such as how initial parameter estimates are guessed and whether and how the design matrix of covariates is mean-centered or orthogonalized to remove collinearities. While re-writing the R rms package logistic regression function lrm I explored several of these issues. Comparisons of execution time in R vs. Fortran are given. Different coding styles in both R and Fortran are also explored. Hopefully some of these explorations will help others who may not have studied MLE optimization and related statistical computing algorithms.

Nov 28, 2024
Frank Harrell
62 min

Adjudication and Statistical Efficiency

classification
decision-making
diagnosis
endpoints
judgment
measurement
medical
design
RCT
accuracy-score
inference
ordinal
subgroup
2024

This article addresses some statistical issues related to adjudication of clinical conditions in clinical and epidemiologic studies, concentrating on maximizing statistical information, efficiency, and power. This has a lot to do with capturing disagreements between adjudicators and uncertainty within an adjudicator.

Oct 17, 2024
Frank Harrell
4 min

The Burden of Demonstrating Statistical Validity of Clusters

classification
data-reduction
diagnosis
medicine
personalized-medicine
subgroup
2024

Patient clustering, often described as the finding of new phenotypes, is being used with increasing frequency in the medical literature. Most of the applications of clustering of observations are not well thought out, not even considering whether observation clustering aligns with the clinical goals. And the resulting clusters are not validated even in a statistical way. This article describes some of the challenges of observation clustering, and challenges researchers to carefully check that found clusters are compact and contain the important statistical information in the variables on which clustering is based.

Oct 6, 2024
Frank Harrell
7 min

Hosting Web Content

computing
2024

This article is about lessons I’ve learned in building and maintaining web sites, with hopefully helpful recommendations.

Sep 29, 2024
Frank Harrell
4 min

Traditional Frequentist Inference Uses Unrealistic Priors

bayes
design
inference
hypothesis-testing
RCT
multiplicity
2024

Considering a simple fixed sample-size non-adaptive design in which standard frequentist inference agrees with non-informative prior-based Bayesian inference, it is argued that the implied assumption about the unknown effect made by frequentist inference (and Bayesian inference if the non-informative prior is actually used) is quite unrealistic.

Jun 10, 2024
Frank Harrell
4 min

Borrowing Information Across Outcomes

bayes
design
RCT
accuracy-score
inference
ordinal
2024

In randomized clinical trials, power can be greatly increased and sample size reduced by using an ordinal outcome instead of a binary one. The proportional odds model is the most popular model for analyzing ordinal outcomes, and it borrows treatment effect information across outcome levels to obtain a single overall treatment effect as an odds ratio. When deaths can occur, it is logical to have death as one of the ordinal categories. Consumers of the results frequently seek evidence of a mortality reduction even though they were not willing to fund a study large enough to be able to detect this with decent power. The same goes when assessing whether there is an increase in mortality, indicating a severe safety problem for the new treatment. The partial proportional odds model provides a continuous bridge between standalone evidence for a mortality effect and obtaining evidence using statistically richer information on the combination of nonfatal and fatal endpoints. A simulation demonstrates the relationship between the amount of borrowing of treatment effect across outcome levels and the Bayesian power for finding evidence for a mortality reduction.

Apr 30, 2024
Frank Harrell
22 min

Proportional Odds Model Power Calculations for Ordinal and Mixed Ordinal/Continuous Outcomes

inference
hypothesis-testing
regression
ordinal
change-scores
design
endpoints
medicine
sample-size
2024

This article has detailed examples with complete R code for computing frequentist power for ordinal, continuous, and mixed ordinal/continuous outcomes in two-group comparisons with equal sample sizes. Mixed outcomes allow one to easily handle clinical event overrides of continuous response variables. The proportional odds model is used throughout, and care is taken to convert odds ratios to differences in medians or means to aid in understanding effect sizes. Since the Wilcoxon test is a special case of the proportional odds model, the examples also show how to tailor sample size calculations to the Wilcoxon test, at least when there are no covariates.

Apr 22, 2024
Frank Harrell
15 min

The log-rank Test Assumes More Than the Cox Model

inference
hypothesis-testing
regression
2024

It is well known that the score test for comparing survival distributions between two groups without covariate adjustment, using a Cox proportional hazards (PH) model, is identical to the log-rank \(\chi^2\) test when there are no tied failure times. Yet there persists a belief that the log-rank test is somehow completely nonparametric and does not assume PH. Log-rank and Cox approaches can only disagree if the more commonly used likelihood ratio (LR) statistic from the Cox model disagrees with the log-rank statistic (and the Cox score statistic). This article shows that in fact the log-rank and Cox LR statistics agree to a remarkable degree, and furthermore the hazard ratio arising from the log-rank test also has remarkable agreement with the Cox model counterpart. Since both methods assume PH and the log-rank test assumes within-group heterogeneity (because it doesn’t allow for covariate adjustment), the Cox model actually makes fewer assumptions than log-rank.

Mar 28, 2024
Frank Harrell
9 min

What Does a Statistical Method Assume?

inference
hypothesis-testing
regression
2024

Sometimes it is unclear exactly what a specific statistical estimator or analysis method is assuming. This is especially true for methods that at first glance appear to be nonparametric when in reality they are semiparametric. This article attempts to explain what it means to make different types of assumptions, and how to tell when a certain type of assumption is being made. It also describes the assumptions made by various commonly used statistical procedures.

Mar 23, 2024
Frank Harrell
22 min

Football Multiplicities

bayes
design
sequential
RCT
accuracy-score
backward-probability
decision-making
forward-probability
inference
multiplicity
prediction
probability
2024

Traditionally trained statisticians have much difficulty in accepting the absence of multiplicity issues with Bayesian sequential designs, i.e., that Bayesian posterior probabilities do not change interpretation or become miscalibrated just because a stopping rule is in effect. Most statisticians are used to dealing with backwards-information-flow probabilities which do have multiplicity issues, because they must deal with opportunities for data to be extreme. This leads them to believe that Bayesian methods must have some kind of hidden multiplicity problem. The chasm between forward and backwards probabilities is explored with a simple example involving continuous data looks where the ultimate truth is known. The stopping rule is the home NFL team having ≥ 0.9 probability of ultimately winning the game, and the correctness of the Bayesian-style forecast is readily checked.

Mar 10, 2024
Frank Harrell, Stephen Ruberg
13 min

How Does a Compound Symmetric Correlation Structure Translate to a Markov Model?

ordinal
prediction
regression
2023
Random intercepts in a model induce a compound symmetric correlation structure in which the correlation between two responses on the same subject have the same correlation…
Dec 3, 2023
Frank Harrell
3 min

Incorporating Historical Control Data Into an RCT

drug-evaluation
bayes
design
drug-development
inference
observational
posterior
prior
2023

Historical data (HD) are being used increasingly in Bayesian analyses when it is difficult to randomize enough patients to study effectiveness of a treatment. Such analyses summarize observational studies’ posterior effectiveness distribution (for two-arm HD) or standard-of-care outcome distribution (for one-arm HD) then turn that into a prior distribution for an RCT. The prior distribution is then flattened somewhat to discount the HD. Since Bayesian modeling makes it easy to fit multiple models at once, incorporation of the raw HD into the RCT analysis and discounting HD by explicitly modeling bias is perhaps a more direct approach than lowering the effective sample size of HD. Trust the HD sample size but not what the HD is estimating, and realize several benefits from using raw HD in the RCT analysis instead of relying on HD summaries that may hide uncertainties.

Nov 4, 2023
Frank Harrell
18 min

Wedding Bayesian and Frequentist Designs Created a Mess

2023
inference
RCT
bayes
design
evidence
multiplicity
posterior
prior
sequential

This article describes a real example in which use of a hybrid Bayesian-frequentist RCT design resulted in an analytical mess after overly successful participant recruitment.

Aug 22, 2023
Frank Harrell
8 min

Ordinal Models for Paired Data

2023
ordinal
hypothesis-testing
inference
regression

This article briefly discusses why the rank difference test is better than the Wilcoxon signed-rank test for paired data, then shows how to generalize the rank difference test using the proportional odds ordinal logistic semiparametric regression model. To make the regression model work for non-independent (paired) measurements, the robust cluster sandwich covariance estimator is used for the log odds ratio. Power and type I assertion \(\alpha\) probabilities are compared with the paired \(t\)-test for \(n=25\). The ordinal model yields \(\alpha=0.05\) under the null and has power that is virtually as good as the optimum paired \(t\)-test. For non-normal data the ordinal model power exceeds that of the parametric test.

Aug 16, 2023
Frank Harrell
10 min

Resources for Ordinal Regression Models

2022
2023
2024
endpoints
ordinal
regression

This article provides resources to assist researchers in understanding and using ordinal regression models, and provides arguments for their wider use.

May 1, 2023
Frank Harrell
20 min

Seven Common Errors in Decision Curve Analysis

decision-making
diagnosis
medicine
2023

I describe seven common errors in decision curve analysis. Avoidance of such errors will make decision curve analysis more reliable and useful.

Mar 18, 2023
Andrew Vickers
7 min

Randomized Clinical Trials Do Not Mimic Clinical Practice, Thank Goodness

generalizability
design
medicine
RCT
drug-evaluation
personalized-medicine
evidence
2017
2023

Randomized clinical trials are successful because they do not mimic clinical practice. They remain highly clinically relevant despite this.

Feb 14, 2023
Frank Harrell
20 min

Biostatistical Modeling Plan

2023
accuracy-score
endpoints
ordinal
collaboration
data-reduction
design
medicine
prediction
regression
validation
bootstrap

This is an example statistical plan for project proposals where the goal is to develop a biostatistical model for prediction, and to do external or strong internal validation of the model.

Jan 26, 2023
Frank Harrell
12 min

How to Do Bad Biomarker Research

2022
big-data
bioinformatics
biomarker
bootstrap
data-science
decision-making
dichotomization
forward-probability
generalizability
medical-literature
multiplicity
personalized-medicine
prediction
principles
reporting
reproducible
responder-analysis
sample-size
sensitivity

This article covers some of the bad statistical practices that have crept into biomarker research, including setting the bar too low for demonstrating that biomarker information is new, believing that winning biomarkers are really “winners”, and improper use of continuous variables. Step-by-step guidance is given for ensuring that a biomarker analysis is not reproducible and does not provide clinically useful information.

Oct 6, 2022
Frank Harrell
15 min

R Workflow

2022
data-science
graphics
r
reproducible

An overview of R Workflow, which covers how to use R effectively all the way from importing data to analysis, and making use of Quarto for reproducible reporting.

Jun 25, 2022
Frank Harrell
15 min

Decision curve analysis for quantifying the additional benefit of a new marker

2022
biomarker
accuracy-score
decision-making
diagnosis
medicine

This article examines the benefits of decision curve analysis for assessing model performance when adding a new marker to an existing model. Decision curve analysis provides a clinically interpretable metric based on the number of events identified and interventions avoided.

Apr 11, 2022
Emily Vertosick and Andrew Vickers
8 min

Equivalence of Wilcoxon Statistic and Proportional Odds Model

2022
2024
endpoints
ordinal
drug-evaluation
hypothesis-testing
RCT
regression

In this article I provide much more extensive simulations showing the near perfect agreement between the odds ratio (OR) from a proportional odds (PO) model, and the Wilcoxon two-sample test statistic. The agreement is studied by degree of violation of the PO assumption and by the sample size. A refinement in the conversion formula between the OR and the Wilcoxon statistic scaled to 0-1 (corcordance probability) is provided.

Apr 6, 2022
Frank Harrell
26 min

Longitudinal Data: Think Serial Correlation First, Random Effects Second

drug-evaluation
endpoints
measurement
RCT
regression
2022

Most analysts automatically turn towards random effects models when analyzing longitudinal data. This may not always be the most natural, or best fitting approach.

Mar 15, 2022
Frank Harrell
10 min

Assessing the Proportional Odds Assumption and Its Impact

2022
accuracy-score
dichotomization
endpoints
ordinal

This article demonstrates how the proportional odds (PO) assumption and its impact can be assessed. General robustness to non-PO on either a main variable of interest or on an adjustment covariate are exemplified. Advantages of a continuous Bayesian blend of PO and non-PO are also discussed.

Mar 9, 2022
Frank Harrell
27 min

A Comparison of Decision Curve Analysis with Traditional Decision Analysis

decision-making
diagnosis
medicine
2021

We compare decision curve analysis and traditional decision analysis to illustrate their similarities and differences.

Dec 27, 2021
Andrew Vickers
7 min

Commentary on Improving Precision and Power in Randomized Trials for COVID-19 Treatments Using Covariate Adjustment, for Binary, Ordinal, and Time-to-Event Outcomes

bayes
covid-19
design
generalizability
inference
metrics
ordinal
personalized-medicine
RCT
regression
reporting
2021

This is a commentary on the paper by Benkeser, Díaz, Luedtke, Segal, Scharfstein, and Rosenblum

Jul 17, 2021
Frank Harrell, Stephen Senn
25 min

Incorrect Covariate Adjustment May Be More Correct than Adjusted Marginal Estimates

2021
generalizability
RCT
regression

This article provides a demonstration that the perceived non-robustness of nonlinear models for covariate adjustment in randomized trials may be less of an issue than the non-transportability of marginal so-called robust estimators.

Jun 29, 2021
Frank Harrell
17 min

Avoiding One-Number Summaries of Treatment Effects for RCTs with Binary Outcomes

2021
generalizability
RCT
regression

This article presents an argument that for RCTs with a binary outcome the primary result should be a distribution and not any single number summary. The GUSTO-I study is used to exemplify risk difference distributions.

Jun 28, 2021
Frank Harrell
10 min

If You Like the Wilcoxon Test You Must Like the Proportional Odds Model

ordinal
hypothesis-testing
2021
accuracy-score
RCT
regression
metrics

Since the Wilcoxon test is a special case of the proportional odds (PO) model, if one likes the Wilcoxon test, one must like the PO model. This is made more convincing by showing examples of how one may accurately compute the Wilcoxon statistic from the PO model’s odds ratio.

Mar 10, 2021
Frank Harrell
6 min

Implementation of the PATH Statement

The recent PATH (Predictive Approaches to Treatment effect Heterogeneity) Statement outlines principles, criteria, and key considerations for applying predictive approaches to clinical trials to provide patient-centered evidence in support of decision making. Here challenges in implementing the PATH Statement are addressed with the GUSTO-I trial as a case study.

Nov 24, 2020
Ewout Steyerberg
19 min

Violation of Proportional Odds is Not Fatal

2020
ordinal
accuracy-score
RCT
regression
hypothesis-testing
metrics

Many researchers worry about violations of the proportional hazards assumption when comparing treatments in a randomized study. Besides the fact that this frequently makes them turn to a much worse approach, the harm done by violations of the proportional odds assumption usually do not prevent the proportional odds model from providing a reasonable treatment effect assessment.

Sep 20, 2020
Frank Harrell
12 min

Unadjusted Odds Ratios are Conditional

2020
generalizability
RCT
regression

This article discusses issues with unadjusted effect ratios such as odds ratios and hazard ratios, showing a simple example of non-generalizability of unadjusted odds ratios.

Sep 13, 2020
Frank Harrell
9 min

RCT Analyses With Covariate Adjustment

2020
drug-evaluation
generalizability
medicine
personalized-medicine
prediction
RCT
regression

This article summarizes arguments for the claim that the primary analysis of treatment effect in a RCT should be with adjustment for baseline covariates. It reiterates some findings and statements from classic papers, with illustration on the GUSTO-I trial.

Jul 19, 2020
Ewout Steyerberg
@ESteyerberg
13 min

Bayesian Methods to Address Clinical Development Challenges for COVID-19 Drugs and Biologics

bayes
RCT
design
drug-evaluation
medicine
responder-analysis
covid-19

The COVID-19 pandemic has elevated the challenge for designing and executing clinical trials with vaccines and drug/device combinations within a substantially shortened time frame. Numerous challenges in designing COVID-19 trials include lack of prior data for candidate interventions / vaccines due to the novelty of the disease, evolving standard of care and sense of urgency to speed up development programmes. We propose sequential and adaptive Bayesian trial designs to help address the challenges inherent in COVID-19 trials. In the Bayesian framework, several methodologies can be implemented to address the complexity of the primary endpoint choice. Different options could be used for the primary analysis of the WHO Severity Scale, frequently used in COVID-19 trials. We propose the longitudinal proportional odds mixed effects model using the WHO Severity Scale ordinal scale. This enables efficient utilization of all clinical information to optimize sample sizes and maximize the rate of acquiring evidence about treatment effects and harms.

May 29, 2020
Natalia Muhlemann MD, Rajat Mukherjee Phd, Frank Harrell PhD
7 min

Implications of Interactions in Treatment Comparisons

RCT
drug-evaluation
generalizability
medicine
observational
personalized-medicine
prediction
subgroup
2020

This article explains how the generalizability of randomized trial findings depends primarily on whether and how patient characteristics modify (interact with) the treatment effect. For an observational study this will be related to overlap in the propensity to receive treatment.

Mar 3, 2020
Frank Harrell
25 min

The Burden of Demonstrating HTE

RCT
generalizability
medicine
metrics
personalized-medicine
subgroup
2019

Reasons are given for why heterogeneity of treatment effect must be demonstrated, not assumed. An example is presented that shows that HTE must exceed a certain level before personalizing treatment results in better decisions than using the average treatment effect for everyone.

Apr 8, 2019
Frank Harrell
6 min

Assessing Heterogeneity of Treatment Effect, Estimating Patient-Specific Efficacy, and Studying Variation in Odds ratios, Risk Ratios, and Risk Differences

RCT
generalizability
medicine
metrics
personalized-medicine
prediction
subgroup
accuracy-score
2019

This article shows an example formally testing for heterogeneity of treatment effect in the GUSTO-I trial, shows how to use penalized estimation to obtain patient-specific efficacy, and studies variation across patients in three measures of treatment effect.

Mar 25, 2019
Frank Harrell
16 min

Statistically Efficient Ways to Quantify Added Predictive Value of New Measurements

prediction
sample-size
validation
accuracy-score
biomarker
diagnosis
medicine
reporting
2018

Researchers have used contorted, inefficient, and arbitrary analyses to demonstrated added value in biomarkers, genes, and new lab measurements. Traditional statistical measures have always been up to the task, and are more powerful and more flexible. It’s time to revisit them, and to add a few slight twists to make them more helpful.

Oct 17, 2018
Frank Harrell
19 min

In Machine Learning Predictions for Health Care the Confusion Matrix is a Matrix of Confusion

data-science
machine-learning
prediction
2018

The performance metrics chosen for prediction tools, and for Machine Learning in particular, have significant implications for health care and a penetrating understanding of the AUROC will lead to better methods, greater ML value, and ultimately, benefit patients.

Aug 28, 2018
Drew Griffin Levy
@DrewLevy
24 min

Data Methods Discussion Site

collaboration
teaching
2018

This article lays out the rationale and overall design of a new discussion site about quantitative methods.

Jun 19, 2018
Frank Harrell
8 min

Viewpoints on Heterogeneity of Treatment Effect and Precision Medicine

RCT
biomarker
decision-making
drug-evaluation
generalizability
medicine
metrics
personalized-medicine
prediction
subgroup
2018

This article provides my reflections after the PCORI/PACE Evidence and the Individual Patient meeting on 2018-05-31. The discussion includes a high-level view of heterogeneity of treatment effect in optimizing treatment for individual patients.

Jun 4, 2018
Frank Harrell
16 min

Navigating Statistical Modeling and Machine Learning

data-science
machine-learning
prediction
2018

This article elaborates on Frank Harrell’s post providing guidance in choosing between machine learning and statistical modeling for a prediction project.

May 14, 2018
Drew Griffin Levy
@DrewLevy
11 min

Road Map for Choosing Between Statistical Modeling and Machine Learning

data-science
machine-learning
prediction
2018

This article provides general guidance to help researchers choose between machine learning and statistical modeling for a prediction project.

Apr 30, 2018
Frank Harrell
11 min

Musings on Multiple Endpoints in RCTs

RCT
bayes
design
drug-evaluation
evidence
hypothesis-testing
medicine
multiplicity
p-value
posterior
endpoints
2018

This article discusses issues related to alpha spending, effect sizes used in power calculations, multiple endpoints in RCTs, and endpoint labeling. Changes in endpoint priority is addressed. Included in the the discussion is how Bayesian probabilities more naturally allow one to answer multiple questions without all-too-arbitrary designations of endpoints as “primary” and “secondary”. And we should not quit trying to learn.

Mar 26, 2018
Frank Harrell
13 min

Improving Research Through Safer Learning from Data

design
evidence
generalizability
inference
judgment
measurement
prior
bayes
2018

What are the major elements of learning from data that should inform the research process? How can we prevent having false confidence from statistical analysis? Does a Bayesian approach result in more honest answers to research questions? Is learning inherently subjective anyway, so we need to stop criticizing Bayesians’ subjectivity? How important and possible is pre-specification? When should replication be required? These and other questions are discussed.

Mar 8, 2018
Frank Harrell
15 min

Is Medicine Mesmerized by Machine Learning?

machine-learning
accuracy-score
classification
data-science
decision-making
medicine
prediction
validation
2018

Deep learning and other forms of machine learning are getting a lot of press in medicine. The reality doesn’t match the hype, and interpretable statistical models still have a lot to offer.

Feb 1, 2018
Frank Harrell
11 min

Information Gain From Using Ordinal Instead of Binary Outcomes

RCT
design
ordinal
dichotomization
inference
precision
responder-analysis
sample-size
2018

This article gives examples of information gained by using ordinal over binary response variables. This is done by showing that for the same sample size and power, smaller effects can be detected.

Jan 28, 2018
Frank Harrell
8 min

Why I Don’t Like Percents

metrics
2018

I prefer fractions and ratios over percents. Here are the reasons.

Jan 19, 2018
Frank Harrell
4 min

How Can Machine Learning be Reliable When the Sample is Adequate for Only One Feature?

prediction
machine-learning
sample-size
validation
precision
accuracy-score
2018

It is easy to compute the sample size N1 needed to reliably estimate how one predictor relates to an outcome. It is next to impossible for a machine learning algorithm entertaining hundreds of features to yield reliable answers when the sample size < N1.

Jan 11, 2018
Frank Harrell
11 min

New Year Goals

2018
2019

Methodologic goals and wishes for research and clinical practice for 2018

Dec 29, 2017
Frank Harrell
7 min

Scoring Multiple Variables, Too Many Variables and Too Few Observations: Data Reduction

variability
data-reduction
2017

This article addresses data reduction, also called unsupervised learning.

Nov 21, 2017
Frank Harrell
6 min

Statistical Criticism is Easy; I Need to Remember That Real People are Involved

RCT
2017

Criticism of medical journal articles is easy. I need to keep in mind that much good research is done even if there are some flaws in the design, analysis, or interpretation. I also need to remember that real people are involved.

Nov 5, 2017
Frank Harrell
6 min

Continuous Learning from Data: No Multiplicities from Computing and Using Bayesian Posterior Probabilities as Often as Desired

bayes
sequential
RCT
2017

This article describes the drastically different way that sequential data looks operate in a Bayesian setting compared to a classical frequentist setting.

Oct 9, 2017
Frank Harrell
13 min

Bayesian vs. Frequentist Statements About Treatment Efficacy

reporting
inference
p-value
RCT
bayes
drug-evaluation
evidence
hypothesis-testing
2017

This article contrasts language used when reporting a classical frequentist treatment comparison vs. a Bayesian one, and describes why Bayesian statements convey more actionable information.

Oct 4, 2017
Frank Harrell
6 min

Integrating Audio, Video, and Discussion Boards with Course Notes

collaboration
teaching
r
reproducible
2017

In this article I seek recommendations for integrating various media for teaching long courses.

Aug 1, 2017
Frank Harrell
15 min

EHRs and RCTs: Outcome Prediction vs. Optimal Treatment Selection

prediction
generalizability
drug-evaluation
evidence
subgroup
EHR
design
medicine
inference
big-data
RCT
personalized-medicine
2017

Observational data from electronic health records may contain biases that large sample sizes do not overcome. Moderate confounding by indication may render an infinitely large observational study less useful than a small randomized trial for estimating relative treatment effectiveness.

Jun 1, 2017
Frank Harrell, Laura Lazzeroni
16 min

Statistical Errors in the Medical Literature

prediction
logic
p-value
validation
bayes
evidence
subgroup
dichotomization
medicine
inference
change-scores
RCT
personalized-medicine
responder-analysis
hypothesis-testing
medical-literature
2017

This article catalogs several types of statistical problems that occur frequently in the medical journal articles.

Apr 8, 2017
Frank Harrell
32 min

Subjective Ranking of Quality of Research by Subject Matter Area

2017

This is a subjective ranking of topical areas by the typical quality of research published in the area. Keep in mind that top-quality research can occur in any area when the research team is multi-disciplinary, team members are at the top of their game, and peer review is functional.

Mar 16, 2017
Frank Harrell
4 min

Damage Caused by Classification Accuracy and Other Discontinuous Improper Accuracy Scoring Rules

prediction
machine-learning
accuracy-score
dichotomization
probability
bioinformatics
validation
classification
data-science
2017

Estimating tendencies is usually a more appropriate goal than classification, and classification leads to the use of discontinuous accuracy scores which give rise to misleading results.

Mar 1, 2017
Frank Harrell
5 min

My Journey from Frequentist to Bayesian Statistics

inference
p-value
likelihood
RCT
bayes
multiplicity
posterior
drug-evaluation
principles
evidence
hypothesis-testing
2017

This is the story of what influenced me to become a Bayesian statistician after being trained as a classical frequentist statistician, and practicing only that mode of statistics for many years.

Feb 19, 2017
Frank Harrell
27 min

Interactive Statistical Graphics: Showing More By Showing Less

survival-analysis
graphics
r
2017

With interactive graphics one can start by showing the most important data features, then drill down to see details.

Feb 5, 2017
Frank Harrell
3 min

A Litany of Problems With p-values

decision-making
bayes
multiplicity
p-value
hypothesis-testing
2017

p-values are very often misinterpreted. p-values and null hypothesis significant testing have hurt science. This article attempts to catalog all the ways in which these happen.

Feb 5, 2017
Frank Harrell
36 min

Clinicians’ Misunderstanding of Probabilities Makes Them Like Backwards Probabilities Such As Sensitivity, Specificity, and Type I Error

specificity
probability
backward-probability
forward-probability
p-value
bayes
conditioning
diagnosis
decision-making
dichotomization
medicine
bioinformatics
biomarker
sensitivity
posterior
accuracy-score
classification
2017

The error of the transposed conditional is rampant in research. Conditioning on what is unknowable to predict what is already known leads to a host of complexities and interpretation problems.

Jan 25, 2017
Frank Harrell
15 min

Split-Sample Model Validation

prediction
bootstrap
validation
2017

The many disadvantages of split-sample validation, including subtle ones, are discussed.

Jan 23, 2017
Frank Harrell
4 min

Fundamental Principles of Statistics

design
measurement
principles
2017
2023

This brief note catalogs what I feel are some of the most important principles to guide statistical practice.

Jan 18, 2017
Frank Harrell
3 min

Ideas for Future Articles

2017

Suggestions for future articles, by readers

Jan 16, 2017
Frank Harrell
2 min

Classification vs. Prediction

prediction
decision-making
machine-learning
accuracy-score
classification
data-science
2017

Classification involves a forced-choice premature decision, and is often misused in machine learning applications. Probability modeling involves the quantification of tendencies and usually addresses the real project goals.

Jan 15, 2017
Frank Harrell
7 min

Null Hypothesis Significance Testing Never Worked

logic
inference
bayes
p-value
hypothesis-testing
inductive-reasoning
2017

This article explains why for decision making the original idea of null hypothesis testing never delivered on its goal.

Jan 14, 2017
Frank Harrell
3 min

p-values and Type I Errors are Not the Probabilities We Need

judgment
inference
likelihood
bayes
multiplicity
p-value
prior
hypothesis-testing
2017

p-values are not what decision makers need, nor are they what most decision makers think they are getting.

Jan 14, 2017
Frank Harrell
12 min

Introduction

2017
principles

Introducing the Statistical Thinking Blog

Jan 13, 2017
Frank Harrell
4 min
No matching items

Talks

Bayesian Thinking

RCT
drug-development
bayes
decision-making
evidence
forward-probability
hypothesis-testing
inference
multiplicity
p-value
posterior
prior
sequential
2024

This presentation covers Bayesian thinking and how different it is from frequentist thinking, with a variety of examples. Unique advantages of Bayesian thinking are emphasized. Some of the topics covered are how frequentism gives the illusion of objectivity by switching the question, an example of frequentist vs. Bayesian answers to a simple question, why α is not the probability of an error, several other contrasts between the two approaches, and multiplicity.

Jan 27, 2025
Frank Harrell

Modernizing Clinical Trial Design and Analysis to Improve Efficiency & Flexibility

RCT
drug-development
bayes
regression
endpoints

This presentation covers several ways to make clinical trials more efficient and to reduce the chance of ending with an equivocal result. Some of the approaches covered are Bayesian sequential designs allowing for study extension if results are promising, not being tied by type I assertion probabilities/α spending, using high-information longitudinal ordinal outcomes, and covariate adjustment.

Dec 5, 2024
Frank Harrell

Ordinal State Transition Models as a Unifying Risk Prediction Framework

endpoints
RCT
ordinal
regression

In this talk I will present a case for the use of discrete time Markov ordinal longitudinal state transition models as a unifying approach to modeling a variety of outcomes for the purpose of estimating risk and expected time in a given state, and for comparing treatments in clinical trials. This model structure can be used to analyze time until a single terminating event, longitudinal binary events, recurrent events, continuous longitudinal data, and longitudinal ordinal responses including multiple events. Partial information can be formally incorporated using standard likelihood approaches without the need for imputation. The model also provides a formal way to assess evidence for consistency of a treatment effect over different outcomes.

Nov 18, 2024
Frank Harrell

Tips for Biostatisticians Collaborating with Non-Biostatistician Medical Researchers

collaboration
endpoints
design
measurement
principles
responder-analysis
medicine
reporting
reproducible
2024

In this talk I contrast consultation with collaboration and discuss various ways to make collaborations most effective. Some key components of effective collaboration are mutual respect, proper division of labor, and basing choices on statistical principles. Special emphasis is given to the importance of biostatisticians being heavily involved in the choice and construction of outcome variables.

Jul 30, 2024
Frank Harrell

Rare Degenerative Diseases & Statistics:
Methods for Analyzing Composite Patient Outcomes

endpoints
RCT
ordinal
regression
2024
bayes
design
measurement
posterior
principles
responder-analysis
sample-size
survival-analysis

In this talk I’ll explain why statistical power is maximized by analyzing the rawest form of clinical trial outcome data, as opposed to analyzing patients at a single time point or reducing rich longitudinal data to time-to-event outcomes. Analyzing all the data longitudinally also provides a formal way to handle missing or partially missing data. A highly flexible longitudinal model will be described in non-mathematical terms. This model is a longitudinal proportional odds logistic model for binary, ordinal, or continuous outcomes, with within-patient serial correlation handled through a simple Markov process in which the patient’s previous visit outcome level becomes a predictor for the outcome in the current time period. This is a discrete time state transition model, also called a multistate model, but extended to have an unlimited number of outcome states as long as they can be ordered.

The model may be fitted with standard software, and special Bayesian modeling software has been written for it.

After the model is fitted, a simple multiplication process converts the transition probabilities into current state probabilities (state occupancy probabilities) to form an intent-to-treat effect that reduces to cumulative incidence of an outcome event if the outcome is binary. The approach accommodates longitudinal outcome scales with clinical event overrides. The primary clinical readout of this longitudinal ordinal model is the average time in condition y or worse, as a function of time and treatment, for any or all outcome levels y.

This approach has the following as special cases:

- Wilcoxon test for comparing two groups on a single-time ordinal or continuous outcome
- Cox proportional hazards model for comparing time-to-event when the event is terminal
- Parametric longitudinal analysis of continuous outcomes
- Recurrent event analysis
- Joint analysis of recurrent nonfatal events and a terminal event
- Estimation of restricted mean survival time when the event is not necessarily terminal

The model also provides the only formal analysis I know for quantifying evidence that a treatment effects different components of the outcome variable differently. For example, one can test whether a treatment lowers mortality by the same relative amount as it lowers disability. The model is particularly well suited for rare degenerative diseases because of formal handling of death, and because its maximum use of information lowers the sample size needed to achieve adequate power, particularly when there are several follow-up visits.

The method will be briefly compared with time-to-event analysis, DOOR (desirability of outcome ranking), and WIN ratio/odds.

A detailed case study of longitudinal ordinal modeling with complete R code may be found at hbiostat.org/rmsc/markov.

Jul 11, 2024
Frank Harrell

Overview of Composite Outcome Scales & Statistical Approaches for Analyzing Them

2024
design
medicine
RCT
ordinal
drug-development
drug-evaluation
endpoints
measurement
regression
responder-analysis

There are many issues surrounding the choice and construction of clinical outcome scales for randomized clinical trials, and several analytical methods from which to choose. This talk overviews some of the issues, and briefly discusses several methods for analyzing longitudinal data: WIN, DOOR, time savings, and ordinal longitudinal models. The discussion is particular relevant to rare and degenerative diseases.

Jan 30, 2024
Frank Harrell

My Big Jump: Founding a Department of Biostatistics

2023
collaboration
medicine
reproducible

For many years biostatistics had been successful at Vanderbilt, but the opportunity to create a department home for biostatistics was too good to pass up. The new department and its support from the School of Medicine leadership made it an attractive place for recruiting new faculty and staff. This talk will cover what made the department attractive, as well as principles upon which the Department of Biostatistics was founded in the Vanderbilt School of Medicine in 2003. These principles include reproducible research and prioritizing collaboration over consultation. Challenges and opportunities of running the department in a growing academic medical center will be discussed, with emphasis on generalizable knowledge that may assist others in starting, sustaining, and enhancing biostatistics groups in their own medical centers.

Mar 20, 2023
Frank Harrell

Controversies in Predictive Modeling, Machine Learning, and Validation

prediction
machine-learning
validation
regression

This talk covers a variety of controversial and/or current issues related to statistical modeling and prediction research. Some of the topics covered are why external validation is often not a good idea, why validating researchers is often more efficient than validating models, what distinguishes statistical models from machine learning, how variable selection only gives the illusion of learning from data, and advantages of older measures of model performance.

Sep 28, 2022
Frank Harrell

R Workflow for Reproducible Biomedical Research Using Quarto

r
data-science
reporting
reproducible

This work is intended to foster best practices in reproducible data documentation and manipulation, statistical analysis, graphics, and reporting. It will enable the reader to efficiently produce attractive, readable, and reproducible research reports while keeping code concise and clear. Readers are also guided in choosing statistically efficient descriptive analyses that are consonant with the type of data being analyzed.

Sep 26, 2022
Frank Harrell

Longitudinal Ordinal Models as a General Framework for Medical Outcomes

endpoints
RCT
ordinal
regression

Univariate ordinal models can be used to model a wide variety of longitudinal outcomes, using only standard software, through the use of Markov processes. This talk will show how longitudinal ordinal models unify a wide variety of types of analyses including time to event, recurrent events, continuous responses interrupted by events, and multiple events that are capable of being placed in a hierarchy. Through the use of marginalization over the previous state in an ordinal multi-state transition model, one may obtain virtually any estimand of interest. Both frequentist and Bayesian methods can be used to fit the model and draw inferences.

Dec 18, 2021
Frank Harrell

Musings on Statistical Models vs. Machine Learning in Health Research

regression
prediction
machine-learning
validation
classification
accuracy-score
data-science

Health researchers and practicing clinicians are with increasing frequency hearing about machine learning (ML) and artificial intelligence applications. They, along with many statisticians, are unsure of when to use traditional statistical models (SM) as opposed to ML to solve analytical problems related to diagnosis, prognosis, treatment selection, and health outcomes. And many advocates of ML do not know enough about SM to be able to appropriately compare performance of SM and ML. ML experts are particularly prone to not grasp the impact of the choice of measures of predictive performance. In this talk I attempt to define what makes ML distinct from SM, and to define the characteristics of applications for which ML is likely to offer advantages over SM, and vice-versa. The talk will also touch on the vast difference between prediction and classification and how this leads to many misunderstandings in the ML world. Other topics to be convered include the minimum sample size needed for ML, and problems ML algorithms have with absolute predictive accuracy (calibration).

Jun 9, 2021
Frank Harrell

Sequential Bayesian Designs for Rapid Learning in COVID-19 Clinical Trials

bayes
RCT
covid-19

Continuous learning from data and computation of probabilities that are directly applicable to decision making in the face of uncertainty are hallmarks of the Bayesian approach. Bayesian sequential designs are the simplest of flexible designs, and continuous learning capitalizes on their efficiency, resulting in lower expected sample sizes until sufficient evidence is accrued due to the ability to take unlimited data looks. Classical null hypothesis testing only provides evidence against the supposition that a treatment has exactly zero effect, and it requires one to deal with complexities if not doing the analysis at a single fixed time. Bayesian posterior probabilities, on the other hand, can be computed at any point in the trial and provide current evidence about all possible questions, such as benefit, clinically relevant benefit, harm, and similarity of treatments.

Besides requiring flexibility in a rapidly changing environment, COVID-19 trials often use ordinal endpoints and standard statistical models such as the proportional odds (PO) model. Less standard is how to model serial ordinal responses. Methods and new Baysian software have been developed for COVID-19 trials. Also implemented is a Bayesian partial PO model (Peterson and Harrell, 1990) that allows one to put a prior on the degree to which a treatment affects mortality differently than how it affects other components of the ordinal scale. These ordinal models will be briefly discussed.

Nov 25, 2020
Frank Harrell

Controlling α vs. Probability of a Decision Error

This presentation clarifies what type I assertion probability α protects against, by making a clear distinction between how often we assert an effect vs. how often we are wrong about an effect. It is argued that “error” should be struck from the phrase “type I error probability”. Frequentist and Bayesian approaches will be briefly contrasted, with an explanation of why it is confusing to mix the two. Terms such as p-values, α, and “false positive” will be attempted to be precisely defined, and subtleties in defining “false positive probability” will be discussed. Then emphasis is placed on multiplicity issues that can occur when analyzing data multiple times, and why such issues do not apply to the Bayesian paradigm. By way of pattern recognition, medical diagnosis, and sequential clinical trial examples it is explained why α loses relevance once data are available.

Nov 13, 2020

Bayes for Flexibility in Urgent Times

bayes
RCT
drug-development
covid-19

Continuous learning from data and computation of probabilities that are directly applicable to decision making in the face of uncertainty are hallmarks of the Bayesian approach. Bayesian sequential designs are the simplest of flexible designs, and continuous learning capitalizes on their efficiency, resulting in lower expected sample sizes until sufficient evidence is accrued due to the ability to take unlimited data looks. Classical null hypothesis testing only provides evidence against the supposition that a treatment has exactly zero effect, and it requires one to deal with complexities if not doing the analysis at a single fixed time. Bayesian posterior probabilities, on the other hand, can be computed at any point in the trial and provide current evidence about all possible questions, such as benefit, clinically relevant benefit, harm, and similarity of treatments.

Jul 1, 2020
Frank Harrell

Fundamental Advantages of Bayes in Drug Development

bayes
inference
multiplicity
drug-development
RCT

This presentation covers the limitations of frequentist inference for answering clinical questions and generating evidence for efficacy. Key to understanding efficacy is understanding conditional probability and its relation to information flow. What type I error really controls is discussed, and it is argued that it is not regulator’s regret. The frequentist and Bayesian approaches for stating statistical results for efficacy assesment are contrasted, and a high-level view of the Bayesian approach is given. A key point is the actionability of the statistical results. Some of the advantages of the Bayesian approach are cataloged, with emphasis on forward-information-flow probabilities that instantly define their own error probabilities. Multiplicity non-issues are discussed.

Apr 27, 2020
Frank Harrell

Casual Inference Podcast

bayes

This interview by Ellie Murray and Lucy D’Agostino McGowan for their Casual Inference podcast recorded 2020-02-26 is titled Getting Bayesian

Apr 23, 2020
Frank Harrell

R for Graphical Clinical Trial Reporting

r
RCT
reporting
safety

For clinical trials a good deal of effort goes into producing both final trial reports and interim reports for data monitoring committees, and experience has shown that reviewers much prefer graphical to tabular reports. Interactive graphical reports go a step further and allow the most important information to be presented by default, while inviting the reviewer to drill down to see other details. The drill-down capability, implemented by hover text using the R plotly package, allows one to almost entirely dispense with tables because the hover text can contain the part of a table that pertains to the reviewer’s current focal point in the graphical display, among other things. Also, there are major efficiency gains by having a high-level language for producing common elements of reports related to accrual, exclusions, descriptive statistics, adverse events, time to event, and longitudinal data. This talk will overview the hreport package, which relies on R, RMarkdown, knitr, plotly, Hmisc, and HTML5. RStudio is an ideal report developement environment for using these tools.

Jan 29, 2020
Frank Harrell

Why Bayes for Clinical Trials?

bayes
RCT
drug-development

This presentation covers the limitations of frequentist inference for answering clinical questions and generating evidence for efficacy. Key to understanding efficacy is understanding conditional probability and its relation to information flow. What type I error really controls is discussed, and it is argued that it is not regulator’s regret. The frequentist and Bayesian approaches for stating statistical results for efficacy assesment are contrasted, and a high-level view of the Bayesian approach is given. A key point is the actionability of the statistical results. Some of the advantages of the Bayesian approach are cataloged, with emphasis on forward-information-flow probabilities that instantly define their own error probabilities. Multiplicity issues are discussed, and a simple simulation study is used to demonstrate the lack of multiplicity issues in the Bayesian context even with infinitely many data looks. Some practical guidance for choosing prior distributions is given. Finally, some examples of joint Bayesian inference for multiple endpoints are given.

Sep 20, 2019
Frank Harrell

R for Clinical Trial Reporting

r
RCT
reporting

Statisticians and statistical programmers spend a great deal of time analyzing data and producing reports for clinical trials, both for final trial reports and for interim reports for data monitoring committees. Point and Click interfaces and copy-and-paste are now believed to be bad models for reproducible research. Instead, there are advantages to developing a high-level language for producing common elements of reports related to accrual, exclusions, descriptive statistics, adverse events, time to event, and longitudinal data.

It is well appreciated in the statistical and graphics design communities that graphics are much better than tables for conveying numeric information. There are thus advantages for having statistical reports for clinical trials that are almost completely graphical. Instead of devoting space to tables, HTML5 and Javascript in R html reports makes it easy to show tabular information in pop-up text when hovering the mouse over a graphical element.

In this talk I will describe R packages greport (using a \(\LaTeX\) pdf model) and hreport (using an html model). knitr and Rmarkdown are used to compose the reproducible reports. greport and hreport compose all figure and table captions. They contain high-level abstractions of common clinical trial reporting tasks to minimize programming by the use. Before showing examples of these report-making packages, I’ll show some of the new graphical building blocks in the Hmisc and rms packages. These new functions make use of the plotly package to create interactive graphics using Javascript and D3.

Sep 13, 2019
Frank Harrell

Bayesian Thinking Podcast

bayes
RCT
drug-development

This interview by Vinay Prasad for his Plenary Session podcast discusses Bayesian thinking, especially about clinical trials.

Aug 7, 2019
Frank Harrell

Regression Modeling Strategies

prediction
validation
accuracy-score
machine-learning
regression

Short course

May 3, 2019
Frank Harrell

Simple Bootstrap and Simulation Approaches to Quantifying Reliability of High-Dimensional Feature Selection

big-data
bioinformatics
machine-learning
multiplicity
prediction
regression
validation

Feature selection in the large p non-large n case is known to be unreliable, but most biomedical researchers are not aware of the magnitude of the problem. They assume for example that setting a false discovery rate makes the results reliable, forgetting about the false negative rate and decades of research showing unreliability of stepwise variable selection even in the low p case. A related problem is the unreliability in the estimate of the effect (e.g., an odds ratio) of a feature found by selecting ‘winners’. This talk will demonstrate some simple bootstrap and Monte Carlo simulation procedures for teaching biomedical researchers how to quantify these problems. One of the bootstrap examples exposes the difficulty of the task by computing confidence intervals for importance rankings of features.

Jul 31, 2018
Frank Harrell

Current Challenges and Opportunities in Clinical Prediction Modeling

prediction
accuracy-score
validation

This talk covers a variety of topics in clinical prediction modeling, with emphasis on quantifying the added value of new predictors.

Jul 4, 2018
Frank Harrell

Using R, Rmarkdown, RStudio, knitr, plotly, and HTML for the Next Generation of Reproducible Statistical Reports

r
reporting
reproducible

The Vanderbilt Department of Biostatistics has two policies currently in effect:
1. All statistical reports will be reproducible
2. All reports should include all the code used to produce the report, in some fashion

We have succeeded with 1. (mainly using knitr in R) and to a large extent with 2. Some biostatisticians have been concerned about interspersing code with the contents of the report. It has also been challenging to copy some PDF report components (e.g., advanced tables) into word processing documents.

Fortunately R and RStudio have recently added a number of new features that allow for easy creation of HTML notebooks that are viewed with any web browser. This solves the problems listed above and adds new possibilities such as interactive graphics that appear in a self-contained HTML file to post on a collaboration web server or send to a collaborator. Interactive graphics allow the analyst to create more detail (e.g., confidence bands for multiple confidence levels; confidence bands for group differences as well as those for each group individually) with the collaborator able to easily select which details to view.

I have made major revisions in the R Hmisc and rms packages to provide new capabilities that fit into the R/RStudio Rmarkdown HTML notebook framework. Interactive plotly graphics (based on Javascript and D3) and customized HTML output are the main new ingredients. In this talk the rationale for this approach is discussed, and the new features are demonstrated with two statistical reports. A few miscellaneous topics will also be covered, e.g. how to cite bibliographic references in Rmarkdown and how to interface R to citeulike.org for viewing or extracting bibliographic references.

For more information see

https://www.r-project.org
https://www.rstudio.com
http://rmarkdown.rstudio.com
http://rmarkdown.rstudio.com/r_notebooks.html
http://yihui.name/knitr
https://hbiostat.org/R/Hmisc
https://plot.ly/r
https://plot.ly/r/getting-started
ggplotly: a function that converts any ggplot2 graphic to a plotly interactive graphic: https://plot.ly/ggplot2

Nov 16, 2017
Frank Harrell

Exploratory Analysis of Clinical Safety Data to Detect Safety Signals

RCT
safety
drug-evaluation

It is difficult to design a clinical study to provide sound inferences about safety effects of drugs in addition to providing trustworthy evidence for efficacy. Patient entry criteria and experimental design are targeted at efficacy, and there are too many possible safety endpoints to be able to control type I error while preserving power. Safety analysis tends to be somewhat ad hoc and exploratory. But with the large quantity of safety data acquired during clinical drug testing, safety data are rarely harvested to their fullest potential. Also, decisions are sometimes made that result in analyses that are somewhat arbitrary or that lose statistical efficiency. For example, safety assessments can be too quick to rely on the proportion of patients in each treatment group at each clinic visit who have a lab measurement above two or three times the upper limit of normal.

Safety reports frequently fail to fully explore areas such as
• which types of patients are having AEs?
• what distortions in the tails of the distribution of lab values are taking place?
• which AEs tend to occur in the same patient?
• how to clinical AEs correlate to continuous lab measurements at a given time
• which AEs and lab abnormalities are uniquely related to treatment assigned?
• do preclinically significant measurements at an earlier visit predict AEs at a later visit?
• how can time trends in many variables be digested into an understandable picture?

This talk will demonstrate some of the exploratory statistical and graphical methods that can help answer questions such as the above, using examples based on data from real pharmaceutical trials.

Jun 8, 2006
Frank Harrell
No matching items

Courses