Statistical Thinking

Bayesian Thinking

Frank Harrell — Tue, 18 Aug 2026 05:00:00 GMT

Janice Pogue Lecture in Biostatistics, Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada 2024-12-06
Center for Biostatistics, Dept. of Population Health Science and Policy, Icahn School of Medicine at Mount Sinai, New York, 2025-03-18.
Department of Biostatistics, Vanderbilt University School of Medicine, 2025-04-23
Medtronic Biostats & Data Science Council, Biostatistics Grand Rounds, 2026-08-18
Slides

Modernizing Clinical Trial Design and Analysis to Improve Efficiency & Flexibility

Frank Harrell — Mon, 10 Aug 2026 05:00:00 GMT

UCLA Cardiology Grand Rounds 2020-10-23 | Video (better video below)
Vanderbilt University Department of Biostatistics 2020-11-18
Vanderbilt Translational Research Forum 2021-11-04 | Video
Consilium Scientific 2024-03-14 | Video and here
Seventh Annual Janice Pogue Lectureship in Biostatistics, Population Health Research Institute, Hamilton, Ontario, Canada 2024-12-05
National Heart and Lung Institute, Imperial College London and Imperial Clinical Trials Unit, Imperial College London, 2026-06-03
Baim Institute for Clinical Research Grand Rounds, 2026-08-10
Slides
Pictures
Video
Baim Institute Grant Rounds Video with updated material 2026-08-10

Important Considerations in Trial Design for Flexibility & Efficiency

RRE 2026: Research RoundTable in Epilepsy, Washington DC 2026-04-30
Shortened version with outcome examples for epilepsy trials
Slides

Parametric Models vs. Empirical Distributions and Ordinal Regression

Frank Harrell — Sun, 09 Aug 2026 05:00:00 GMT

Background

In modeling data we frequently need to estimate more than a mean. We may want to estimate quantiles, dispersion, or tail probabilities. These require model assumptions to be correct when the model is a smooth parametric one. Even when estimating a mean, for inference to be accurate we need distributional assumptions we make to be satisfied. The central limit theorem offers no protection especially when the underlying distribution is asymmetric.

Consider a continuous, discrete, or mixed-type random variable having a cumulative distribution function . The empirical cumulative distribution function (ECDF) defined below is the nonparametric maximum likelihood estimator of from a sample of values .

where the indicator function is if is true, otherwise. The summation is just a count of the number of data values at or below . The ECDF can be thought of as a cumulative histogram with no binning. It is uniquely defined and requires no choices to be made in its calculation.

There is a question of whether it is worth the trouble to fit a parametric distribution instead of relying on the ECDF. Does a parametric fit gain enough precision to offset the risk that the parametric distribution family may not adequately fit the data? Is it fair to think of the ECDF as essentially equivalent to a parametric estimator with several parameters capturing location, spread, skewness, heavy-tailedness, bimodality, and more? In other words does the variance of the ECDF mimic fitting a good many parameters to the distribution ? How many parameters?

ECDFs and parametric distributions that don’t have covariates are intended to be used for homogeneous samples and populations. To allow for heterogeneous tendencies in we introduce covariates . Ordinal semiparametric regression models in the cumulative probability model (CPM) family are covariate-shifted ECDFs so they inherit the distribution-free advantages of the ECDF. A CPM may be stated as

where is some smooth increasing function, is a vector of intercepts, is a vector of covariate values, are regression coefficients, and means “conditional on” or “given”. Semiparametric models are distribution-free on the left side of the model and are parametric on the right side. Fortunately, the right side can be made arbitrarily flexible through the use of spline functions and interaction terms, with the only limitations coming from the effective sample size.

Ordinal CPMs could be called parametrically shifted (by ) nonparametric models.

Common choices for are the logistic and normal cumulative distribution functions. There are as many intercepts as there are distinct values, less one. This flexible vector of intercepts is what allows CPMs to fit any distributional shape. Whereas parametric models make absolute distributional assumptions, semiparametric ones such as CPMs make only relative distributional assumption, i.e., whatever shape has, for different values of are assumed to be parallel. Parametric models assume that vs. is a straight line in addition to assuming the different straight lines for different s are parallel.

When there are no covariates or , the intercepts are just the -inverses of the ECDF. So the ECDF is equivalent to a CPM with no covariates; for any when . Accurate estimation of cumulative probabilities using CPMs goes hand-in-hand with accurate estimation of using an ECDF. One of our side aims is to estimate the effective number of parameters that the intercepts really involve.

The intercepts are order-restricted, making them have far lower effective degrees of freedom than apparent at first. The order restriction also explains why one can fit a CPM with hundreds of thousands of intercepts extremely quickly, making CPMs work well for continuous even for very large .

The R rms package orm function can fit a model with and no ties in (299,999 intercepts) and 10 covariates in 2.5 seconds.

Binary logistic regression is a special case of CPMs. It is virtually impossible to estimate accurately unless you can estimate accurately. This requires an accurate estimate of the (single) intercept, which equates to estimating a simple probability with a proportion. When is unrestricted other than being forced to be in the interval , a confidence interval for has maximum width when . So to have a good estimate of for all possible requires a good estimate when . In this case, the sample size needed to estimate to within a margin of error of at the 0.95 confidence level is . To have a margin of error of requires .

The Semi-Nonparametric (SNP) Density Family

Consider a continuous variable with a continuous cumulative probability distribution . To compare the variance and mean squared error of the ECDF with a parametric family used to estimate , we need a family that incorporates a systematic progression from 2 parameters (mean and variance) to 3, 4, 5, … . Such a family is the semi-nonparametric (SNP) density of Gallant & Nychka (1987). This is a squared polynomial multiplying a standard normal kernel,

where is the standard normal density, , and

is a normalizing constant. When this reduces exactly to the ordinary Gaussian , and each additional polynomial term adds one more shape degree of freedom — skewness, kurtosis, multimodality, and so on — while the squaring guarantees the density stays non-negative everywhere. As , this sequence is dense in a broad class of smooth densities, which is exactly why it’s a natural tool for interpolating between “fully parametric” and “fully nonparametric” to study how many parameters it takes to approach ECDF-level behavior.

To fit an SNP distribution to data, the SNP parameters are estimated using maximum likelihood.

Simulating Accuracy of for Population Median

We simulate data from two very different continuous distributions, for varying sample sizes. The first distribution is the standard normal (Gaussian) distribution, which is the simplest SNP distribution (). The second is an SNP distribution with for suitably chosen to result in a skewed bimodal distribution which will be plotted later. For each of 1000 samples for each and each data generating distribution we compute the ECDF and fit 7 SNP distributions, for . These SNP fits have parameters.

From each estimator of we estimate the accuracy of the estimated value at a single point for simplicity: the population median . So we know that the true at that point is . We know that the ECDF is exactly unbiased for every distribution, so the expected value of ECDF = . The variance of the ECDF at the true median is also known. It is which is also the mean squared error.

The following code was written by Claude Sonnet 5 after extensive interactions.

Code

Statistical Models Answer the Fundamental Clinical Question and Provide Clinical Trial Estimands

Frank Harrell — Mon, 27 Jul 2026 05:00:00 GMT

Goals and estimands should flow from the study design while respecting the lack of exchangeability of the participants and leading to estimators with data-verifiable assumptions.

Statistical models are approximations of reality. It is not rational to fear transparent assumptions required by them. Avoiding models will almost surely lead to worse results.

It is impossible to model an individual patient’s outcome from cross-sectional or parallel-group designs. The best we can do is to model outcome tendencies.

Background

Before making a case for the most relevant estimands for traditional clinical trials, we need to introduce estimands, the roles for statistical models, and how the most common randomized clinical trial (RCT) design works.

Estimands

Features and estimands should be interpretable in the real world, meaning they are based on an observable process. — A Bühler, RJ Cook, JF Lawless 2023

This article considers the key quantities being estimated, i.e., key estimands, in an RCT. It does not consider the estimands framework per se, which often focuses on estimands in complex situations such as non-adherence, dropouts, and unscheduled treatment crossover. This article focuses on the more classical intent-to-treat analysis of traditional outcomes, without intending to answer “what if” questions such as “what is the treatment effect we expect had all patients adhered to assigned treatment or had none of them died before their final clinic visit?”. Non-adherence is averaged out, and an intent-to-treat analysis estimates differences in outcomes depending on initial treatment assignment. For an RCT treatment effectiveness estimate to transport to clinical practice, adherence levels observed in the RCT may need to mimic those in clinical practice. Attempts to adjust for differing levels of adherence requires making untestable statistical model assumptions including assumptions about reasons for non-adherence.

The types of estimands considered here include both absolute and relative measures. Absolute measures include differences in

means, medians, other quantiles, pseudomedians
absolute risks
cumulative incidences
mean or median time to an event, e.g. gain in life expectancy

Relative measures include

odds ratios
risk ratios (more context-dependent than odds ratios)
hazard ratios
ratios of means

Statistical Models

As long as a model appears to describe the relevant aspects of the world satisfactorily, we may continue, cautiously, to use it; when it fails to do so, we need to search for a better one. — AP Dawid, 2000

Statistical models are approximations of unknown data generating processes. The fact that they are not perfect does not imply they should be avoided. Avoiding data models results in oversimplification, lack of transportability of clinical trial findings to clinical populations, reduced utility for individual patient decision making, and overuse of impossible-to-actualize counterfactual thinking.

To find the most efficient estimator of a parameter, i.e., an estimator having the highest probability of being close to the true parameter value, one must have a data generating model that includes a distribution for the response variable. For that distribution one can derive an optimum estimator in terms of performance measures such as precision and root mean squared error. The data model also allows for statistical inference, including the calculation of uncertainty intervals.

The data generating model does not have to be perfect. When the main effect of interest is an experimental condition that was randomized, the data model is allowed to be even more imperfect.

Consider the oldest statistical regression model, the linear model, named from the fact that it is linear in the parameters. Suppose that the outcome variable is continuous, and conditional on a vector of baseline descriptors has a normal distribution. Let denote a vector of baseline descriptors, a vector of regression coefficients corresponding to them, and denote their inner product, i.e., , also called the linear predictor. The linear model may be stated as where is a residual or error term having a normal distribution with mean zero. The linear model is flexible only in allowing to be expanded to contain nonlinear and non-additive terms (interactions); it is inflexible with regard to the normality assumption for and the need for to already be properly transformed so that effects of the baseline variables are largely additive and the error term is additive. Parametric models for continuous are very sensitive to these transformations.

Models such as the ordinary multiple regression model above which have an error term have one distinction: When an important predictor is left out of the vector, its effect is absorbed into the error term by making the residuals larger. When one of the ’s such as treatment is uncorrelated with the omitted variable (i.e., treatment assignment is balanced with respect to the omitted variable), the treatment effect estimate is not biased by this omission. Only power is lost, due to an increase in residual variation.

Statistical models are more general, flexible, and robust than many researchers believe (see also this). Consider a very general statistical model that is part of the cumulative probability model (CPM) family of ordinal semiparametric models. CPMs are stated in terms of a standardized underlying cumulative probability distribution , such as the normal distribution (probit ordinal model), the logistic distribution (proportional odds model), and the Gumbel distribution (proportional hazards model). The ordinal outcome variable may be continuous, discrete, a discrete/continuous mixture, or binary. Note that no assumption is made that follows the distribution . is chosen because it is monotonically increasing. Semiparametric models do not make absolute distributional assumptions, only relative ones. is the scale on which the effects are shifted. For example suppose a probit ordinal model is used and the only is treatment A vs. B. Then is the cumulative normal distribution, and the model assumes that whatever shape the cumulative distributions have for A and B, the normal inverse of their CDFs (cumulative distribution functions) are parallel. The normal inverse CDFs do not need to be linear, which is required for a standard parametric regression model. For a Cox semiparametric proportional hazards survival model we want survival curves to be parallel but do not require them to be linear (as a parametric Weibull model would).

See this for a gentle introduction to ordinal models and this for more about their unifying powers.Discrete/continuous mixtures include continuous responses with clumping at zero and continuous variables with clinical event overrides at the extremes. Some values may be censored.

The CPM may be stated in terms of exceedance probabilities: where is any of the distinct values of the outcome , means “given” or “conditional on”, and denote a vector of intercepts. There are as many intercepts as there are distinct -values occurring in the data, less one. When the baseline variables have no effect, and the CPM is just a restatement of the empirical cumulative distribution function (ECDF), which is the nonparametric maximum likelihood estimator of the cumulative distribution of . When , a CPM can be thought of as a shifted ECDF. CPMs are semiparametric because they are parametric with respect to but make no distribution assumption about for any specific . When has only two distinct levels and is the logistic function , the CPM is the binary logistic model. To make the model more nonparametric with regard to the effects one can add regression splines and interaction effects inside so as to not assume linearity or additivity on the linear predictor scale. To make the model parametric, just make be a smooth function that does not adapt to the data. For the linear model .

These unitless intercepts pertaining to cumulative probabilities should not be confused with the traditional linear model intercept on the -scale.

CPMs work perfectly well for continuous . The R orm function in the rms package has no limit on the number of intercepts. These intercepts encode the ECDF of when and in the Cox proportional hazards model represent where is the underlying survival curve, which is a step function.

Ordinal models in the CPM family have the following as some of their special cases: sample mean and quantiles, Pearson’s test, -test (two-sample and paired), ANOVA, linear regression, Wilcoxon test, Wilcoxon signed-rank test, Kruskal-Wallis test, ECDF, Kaplan-Meier estimator, Turnbull estimator for interval censored , logrank statistic, Cox proportional hazards model, and accelerated failure time models. See this for how to analyze the special cases with a CPM.

CPMs are robust to outliers in and the parameter values ( and ) do not change when any monotonic increasing transformation is made to . and computed on are identical to estimates computed on . CPMs use only the ordering of and have such wide applicability and fewer assumptions that they should be on the front line of model choices for RCTs.

Until one uses a fitted CPM to estimate the mean of

CPMs provide a wide number of clinical readouts including cumulative probabilities, survival curves, restricted mean survival time, probability of an outcome at a specific level , effect ratios, and covariate-specific differences in means and quantiles, along with uncertainty intervals.

Unlike the linear model with its , and omitted predictor can distort the effect of an even with perfect balance in the omitted covariate across treatments. This is related to non-collapsibility of odds and hazard ratios which has inappropriately been used to criticize these ratios. Non-collapsibility is of little concern if alternatives to statistical modeling make matters even worse. Covariate-adjusting for some baseline variables is better than not covariate-adjusting for any.

Statistical models readily account for differential treatment effect (treatment baseline covariate interactions). Interactions are usually covered up in the causal inference world.

RCT design

A clinical trial gains its value, not from representativeness, but from the internal randomisation that ensures that a comparison of between its treated and untreated groups is indeed a comparison of like with like, and that valid probability statements can be made about likely differences, to enforce internal validity. — AP Dawid & S Senn

Consider a comparison of the therapeutic effectiveness of treatments A and B when a given patient will receive only one of the treatments, at random. This is a parallel-group randomized clinical trial (RCT) design. The selection process involved in this classic design is a triple filtration, at least:

The RCT is advertised, or patients are asked by medical staff about their interest in participating.
Patients volunteer for participation, for a variety of reasons.
These volunteers are screened to see if they meet the study’s inclusion criteria and don’t meet any of the exclusion criteria.
Those passing the previous screens go through informed consent to be randomized.
Between the time the formal consent is signed and actual randomization takes places, patients can change their minds and withdraw from the study. Since randomization hasn’t happened yet there are no significant harms to the study other than a reduction in sample size.
Once a patient is randomized, A or B begin and follow-up starts.
After randomization, patients may drop out and revoke their consent to even be contacted again. This may happen before or after treatment actually starts. Also, some patients may not fully adhere to the assigned therapy. To deal with these complexities without using complex uncheckable analyses, most RCTs base their pivotal analyses on intent-to-treat or “analyze as you randomize”.

RCTs exist to estimate what happens to patients after they pass through the multi-phase filtration process described above. It is a mistake to think that an RCT ever intended to estimate population average responses for a treatment; RCTs do not use random sampling nor probability sampling from a clinical population. The difference between A and B is the focus. Depending on how differences are estimated, they may or may not unbiasedly estimate population quantities. The use of marginal estimators outside of the linear model framework assumes either that the RCT selected patients completely at random from a population (i.e., patients are coerced to take risky experimental therapies) or they are selected under probability sampling and we know the selection probabilities. These never apply to an RCT. Marginal treatment effects (especially when absolute measures are being used) computed on patients in an RCT do not transport to a population and are difficult to interpret even within the RCT sample (see this and this).

RCTs do not mimic clinical practice nor should they.

A very important point routinely missed by practitioners of causal calculus is that even with strict inclusion criteria, most RCTs still randomize patients with a wide prognostic spectrum. This heterogeneity of outcome tendencies means that patients are not exchangeable, and when the underlying statistical model is not linear, marginal estimates of absolute effects may not apply to any patient in the trial, and they cover up extreme heterogeneity of absolute treatment effects. In the so-called age of precision/personalized medicine, marginal estimates are instead highly depersonalized.

Conceptualizing Clinical Questions in an RCT

… the meaningfulness of a purportedly scientific theory, proposition, quantity, or concept is related to the implications it has for what is or could be observed, and in particular, to the extent to which it is possible to conceive of data that would be affected by the truth of the proposition or the value of the quantity. When this is the case, assertions are empirically refutable and are considered “scientific.” When this is not so, they may be branded “metaphysical.” I argue that counterfactual theories are essentially metaphysical. This in itself might not be automatic grounds for rejection of such a theory, if the causal inferences that it led to were unaffected by the metaphysical assumptions embodied in it. Unfortunately, this is not so, and the answers the approach delivers to its inferential questions are seen, on closer analysis, to be dependent on the validity of assumptions that are entirely untestable, even in principle. This can lead to distorted understandings and undesirable consequences. — AP Dawid, 2000

For inference about the effects of causes, a straightforward “black box” decision-analytic approach, based on models and quantities that are empirically testable and discoverable, is perfectly adequate. — AP Dawid, 2000

Imagining what we could know or do, if only we had more information that we actually do have, gets us nowhere. — AP Dawid & S Senn

There are at least three ways to conceptualize the fundamental clinical question, the first being a counterfactual approach.

For a patient who received treatment A, would she have gotten a different outcome had she received treatment B instead?
Does a group of patients randomized to A have a different mean response than a group randomized to B?
Do patients starting at the same place except for treatment end up in the same place?

Outside of crossover designs (not covered here), option 1 is not tied to any data so its assumptions and analytical methods based on it are unverifiable. This is an individual-patient counterfactual question. This question does not need to be answered to choose a treatment prospectively and is not identified by parallel-group RCT data under any analytic framework.

Because of heterogeneity of ages, severity of disease, and other prognostic factors across patients within treatment group, option 2 is not as relevant to clinical practice at it seems. The question is stated as if patients all given the same treatment are exchangeable. They are not.

The Fundamental Question in an RCT

I submit that option 3 above, when made more precise, is the main question of interest in parallel-group RCTs. It uses standard or flexible statistical models and is completely linked to data, so the fit of the model needed to answer the question can be checked. But we need to more precisely word this option. We could wish for what many causal inference advocates emphasize: individual response to treatment. But without a crossover study we cannot begin to discuss individualized responses. And we must recognize that patient responses are not intrinsic because they have a strong random component (sorry about that, potential outcomes). When assessed frequently enough you will see a variety of responses within one patient. So thinking about patient-specific treatment differences in a parallel-group design is not fruitful. Instead, emphasize (1) treatment differences within patient types and (2) outcome tendencies instead of individual outcomes. The fundamental clinical question becomes:

For a given patient type, what is the difference in outcome tendencies for such patients assigned treatment A vs. those assigned treatment B?

This question is completely concordant with a parallel group design, and is completely informed by the observed data. We have to be clear on patient type. By that we mean a patient having a combination of characteristics contained in . For patient characteristics not contained in the study’s measured covariates , we mean the comparisons to apply in the case where all other important (prognostic) characteristics are equal. So the fundamental question can be re-phrased as the following. Keep in mind the statistical model’s role as an interpolator to estimate outcome tendencies for a large variety of covariate combinations (e.g., age=60, sex=female, severity of disease=19) even when their exact values are not observed in the RCT data.

Fundamental Clinical Question

For patients with a given combination of characteristics , if we could assign treatment A to one such patient and B to another patient of the same type—all else about them being equal—what is the difference in their outcome tendencies?

Randomization allows us to include “all else being equal” and to read this as the effect of assigning treatment, not merely of observing outcomes of patients who happened to get these treatments. I deliberately say assign rather than give or receive. A clinician can assign a treatment; she cannot compel a patient to adhere to it. This is not a minor wording choice — it is what ties the patient-type estimand to the intention-to-treat principle: the target throughout this article is the effect of the assignment, not of treatment actually received.

As discussed here, when one wants to estimate a treatment effect among adherers, causal calculus is necessary.

Consistent with each patient in a parallel-group design getting only one treatment, the estimand arising from this phrasing of the fundamental question is a patient-type estimand.

The tendency measure will have to be chosen, e.g., differences in means, pseudomedians, medians and other quantiles, risk, cumulative incidence, restricted mean survival time, time spent in good functional status, or relative tendencies such as odds or hazard ratios.

CPMs can provide both absolute and relative measures.

A key to understanding the fundamental question is understanding the dimensionality of the parameters involved for inference and estimation. This will be described below.

Statistical Estimands to Answer the Fundamental Clinical Question

Think of a model as an interpolator and a smoother that accounts for easy-to-explain outcome heterogeneity (from patients being unlike each other) and that provides useful and parsimonious estimates of treatment effects that apply to patients. These estimates are individualized to within the resolution of measured patient characteristics. The choice of the general model structure is dictated by the study design and must be concordant with it.

General Model and Its Underlying Model Parameters

For a binary, ordinal, continuous, or mixed univariate outcome variable consider the following semiparametric model. Define the following sets of baseline characteristics:

: patient characteristics other than treatment that are thought to be predictive of , collected in the RCT, and used for covariate adjustment
: treatment assignment ( for A, for B)
: subset of that may interact with treatment (differential effect with respect to )
baseline variables that are predictive of but were not included in

The most important variable to include in is a time zero assessment of . If this is not the severity of the disease being studied, then severity of disease at baseline is also important to include.

Now consider the CPM semiparametric regression model: where is a suitably chosen distribution family.

Often is the logistic or Gumbel CDF to make parameters more interpretable as odds or hazard ratios. Deviance can be used to choose the best fitting link function ().

On the fundamental model-building parameter scale, the treatment effect is when is absent or . Otherwise the A B treatment effect on the linear predictor scale (e.g., log odds) is . This treatment effect is a “rung-2” quantity in Judea Pearl’s causal calculus. That’s because and are estimated from a randomized trial in which Pearl’s do-operator was actually executed; it is genuinely causal and not just associational. Pearl’s rung 3 of causation, the counterfactual rung, does not ask what the treatment does for a patient of a certain type, but what would have happened to a certain patient, already observed under one treatment, had they received the other. To draw actionable causal conclusions from a clinical trial, one can use a purely non-counterfactual decision-theoretic approach as so elegantly argued by Philip Dawid and Dawid and Senn. That approach is more in line with what is presented here.

Technical detail: if the model is extended to allow for -dependent effects, e.g., time-varying hazard ratios in a proportional hazards model, the treatment effect on the linear predictor scale will be a function of time and will not be an intent-to-treat causal estimand. The model will have to be translated e.g. to cumulative incidence or mean restricted survival time to get intent-to-treat estimands.

The patient-type estimand advocated here is not a retreat from causality. It jumps off of Pearl’s hierarchy one step short by not trying to deal with unobservables.

Since is unknown we have to pretend operationally that . So our estimates will be marginalized over . Interestingly, if there are repeated measures for each patient, one can add patient-specific random intercepts to the model to absorb the effect of unmeasured covariates . Having a baseline version of in the model often accomplishes the same thing.

Only a minority of RCTs are sufficiently sized to have excellent power to detect a true minimum clinically important difference even in the absence of interactions. Almost no RCTs are sized to allow for differential treatment benefit. So and will usually be omitted, i.e., interactions are assumed to be absent, and if they are not, treatment effects are averaged over levels of interacting factors. Note that all of these complexities, often used to criticize models by those not liking regression models, are not well handled by any of their alternatives. The beauty of models is that they expose assumptions, provide efficient and accurate results when assumptions are reasonably well satisfied, and the assumptions can be checked with data and model enhancements made when there is a meaningful lack of fit. Were an analyst to fit interaction terms that are not well-supported by data, uncertainty intervals would be appropriately wide to reflect this.

Alternatively, Bayesian shrinkage priors or penalized maximum likelihood estimation can be used to account for interactions without overfitting. In this large RCT example the optimum penalty for interaction terms was indicating amazing constancy for the odds ratio for treatment.

Dimensionality is a Key to Understanding

To fully understand what is going on with statistical models used to infer treatment effectiveness and to estimate patient-type-specific effectiveness one must understand the dimensionality of the problem and how that dimensionality depends on the goal. Think of dimensionality as degrees of freedom and remember that knowing and accounting for the actual degrees of freedom allows for honest assessment of uncertainties and avoidance of cherry picking (including the neverending search for subgroups). Suppose that contains columns and has columns, . describes differential treatment benefit (interactions) on the linear predictor (LP, e.g., log odds) scale. When , no interactions are allowed. Here are the dimensionalities involved with different general and specific goals.

General Goal	Patient Type	Dimensionality	Comment
A B		1	effect for a specific value of interacting factor
A B	all		does an effect exist for any patient type
Magnitude of B-A effect on a clinical scale			e.g. absolute treatment benefit for patient type

Inference for nonzero effects can be done strictly on the LP scale, because absolute differences are zero if and only if relative (LP) differences are zero. Estimation of magnitudes on meaningful scales (absolute risk reduction, gain in life expectancy, etc.) carries along parameters used to estimate and effects.

Clinical Scale Treatment Effectiveness Estimands

Once a flexible and efficient Bayesian or frequentist model is fitted on the RCT data, one immediate gets differences on the linear predictor scale. Except for the case where is the cumulative normal distribution and the cumulative probability intercepts ( being the residual standard deviation), where the model is parametric and instantly provides differences in mean , one is usually interested in translations of the model to a scale that is more relevant to patients and clinicians. Suppose the fitted model is with denoting the linear predictor and parameter estimates (e.g., maximum likelihood estimates) inserted for the Greek letters. Transformations to clinical readouts can be written as . When this is nonlinear, the effects of baseline covariates and will not be separable from the effect of even in the absence of interactions. For that reason, estimates coming from nonlinear functions of must be covariate-specific. See this for concrete examples of .

As examples, a special case of CPMs is the proportional odds ordinal logistic model. It directly estimates cumulative probabilities, and from these one can solve for which intercept yields any given cumulative probability. Then the value of that corresponds to that intercept is looked up and this is the estimate of a quantile of . Successive differences of the cumulative probabilities are cell probabilities, and a sum of these probabilities weighted by all the distinct values estimates the mean of . This is done for each treatment, and the needed differences computed. Bayesian modeling provides exact uncertainty intervals for such differences using the posterior samples already drawn. For frequentist modeling one uses the bootstrap or the -method to get confidence intervals. For cumulative incidences or survival probabilities, so things are particularly simple.

To reinforce the need for covariate-specificity, remember that low-baseline-risk patients have little room to move, and those at moderately high risk have the most room for risk movement. When the clinical scale is absolute risk, this means that risk magnification must be taken into account. RCTs as typically reported pretend this doesn’t exist, and only provide crude average marginal risk differences that not only may not apply to future patients but may not even apply to any patient who was randomized.

How should the results of clinical trials be reported when results are covariate-specific? Two approaches come to mind:

choose representative covariate combinations and provide treatment effect estimates and uncertainties for each
estimate differences for all patients in the RCT and show the entire distribution

When there are continuous predictors, typically every person in the RCT will have different estimated values. So we have estimands and estimates for patients! This is not a problem, and a high-resolution histogram showing this distribution will honestly portray expected treatment effects.

When interactions are present, the true dimensionality of these estimates is one plus the number of columns in . When there are no interactions, there is effectively only one effectiveness parameter for inference even with manifestations of differences. This is because the differences (e.g., absolute risk reductions) are zero if an only if . For a proportional odds model, for example, the treatment odds ratio is 1.0 if an only if all estimated risk reductions are 0.0. So evidence for effectiveness concentrates the power into a one degree of freedom test when there are no interactions.

Here is an example of the distribution of estimated absolute risk reductions from the 41,000 patient GUSTO-I trial of tissue plasminogen activator vs. streptokinase for acute myocardial infarction. A binary logistic model was used to estimate 30-day mortality. Estimates are made for all patients in the trial, first setting treatment to streptokinase then to -PA. First, show the estimated 30-day mortality under -PA vs. under streptokinase. Clearly apparent is that the RCT is dominated by a very large number of low-risk patients for whom absolute risk reduction is minimal.

Next examine a histogram of the estimated patient-type-specific absolute risk reductions.

Vertical arrows depict the median (left) and mean (right) absolute risk reductions. The marginal difference in risks is not representative of patients in the trial, and gives heavy weight to the minority of high-risk patients. The median risk reduction (not estimable without a model) is more representative of what patients should expect to benefit.

Either or both of the graphs are excellent effectiveness summaries that fully document the variety of prognoses of the randomized patients which leads to the variety of effectiveness estimates when using an absolute scale.

A simple nomogram allows patient-type-specific estimates of absolute risk reduction to be easily calculated.

Celebrate Incorrect Statistical Models

Far better an approximate answer to the right question, which is often vague, than the exact answer to the wrong question, which can always be made precise – John Tukey

One person’s way of doing something is usually better than another person’s way of not doing it.

Critics who worry about wrong models make things worse by their fix.

A critic who says “your logistic model with included could be distorted by an omitted variable ” is making a claim that requires knowing exists and roughly how it behaves — information neither you nor the critic’s alternative estimator actually has. If their proposed fix is an unadjusted or marginal analysis, that estimator doesn’t use either. So the critique can’t be “I have more information and can do better with it” — nobody has . The critique, stripped of its rhetorical force, reduces to: given the same information you have, I’d process it differently. That’s a legitimate methodological debate, but it’s a different and much narrower claim than “your model is unreliable because of what it’s missing.” The right response is to ask what their alternative actually buys you given the same ignorance of .

Proponents of causal calculus criticize assumptions made by nonlinear statistical models (despite often being unaware of distribution-free semiparametric ones) and seek to constantly invent new approaches such as doubly-robust estimators, targeted maximum likelihood estimation, machine learning-based incorporation of covariates, and -computation. Critics of more standard statistical models tend to put a lot of emphasis on bias and little emphasis on mean squared errors or individual-patient clinical decision making. Users of -computation free themselves of assumptions about treatment covariate interactions and robustify inference by effectively allowing for all possible treatment covariate interactions but then sweeping such interactions under the rug by estimating marginal (non-covariate-conditional) treatment effects. This is often done by fitting separate logistic regressions in the two treatment groups, estimating the treatment-related risk difference for each patient, then averaging these differences over all patients. The incredible instability in per-patient estimates is averaged out. This results in reduced variance on the risk scale but at the great cost of changing the question. This approach, besides hiding whatever interactions may be present, jettisons some of the richest information in a study: the prognostic spectrum and its relationship to absolute treatment benefit, demonstrated in the graphs above. This has direct implications in how treatments are used in practice. Expensive or dangerous treatments should be reserved for patients with larger absolute benefits. Marginal estimates do not transport to clinical populations and do not compare in value to patient-type-specific estimates.

Counterfactual-based attempts at individual patient decision making have led to nothing but confusion.

Missing Covariates

One of the criticisms of nonlinear statistical models is that due to non-collapsibility, failing to adjust for the hidden outcome heterogeneity caused by unmeasured covariates distorts estimates and inference related to parameters for treatment and measured covariates. Let’s study this in a very simple situation where the response variable is binary, there are two treatments ( for A, for B), one binary measured covariate , and one binary unmeasured covariate . Suppose the true model is where . Make these correlations zero: vs. , vs. , vs. . Regression coefficients are all 1.0. Consider two approaches:

modeler: fit the above model without the unavailable and proceed as usual
critic: relax assumptions using any estimation technique desired with the goal of estimating the effect of marginalized over both and . -computation can be used for variance reduction for the final contrast. Patient-type resolution is traded for a sample-level quantity.

Here are infinite results for odds ratios(ORs).

In the table “Uses neither nor ” refers to the marginal risk difference estimand, not the variance estimator.

Table 1: Relative scale (log-odds-ratio)

Estimate	Uses	log-OR	Bias vs. true 1.000
True effect	X and U	1.000	—
Modeler’s additive model (T + X, U omitted)	X only	0.947	−0.053
Critic, fully marginal	neither X nor U	0.899	−0.101

Using the one available covariate with missing — cuts the bias from omitting relevant covariates roughly in half (0.101 down to 0.053). This is not from correctly specifying anything about ; it is simply what conditioning on measured predictors does to a non-collapsible measure.

Here are the results for the absolute risk difference due to treatment.

Table 2: Absolute scale (risk difference)

Estimate	Value	Target	Bias
Modeler,	0.2323	0.2311 (true, -averaged)	+0.0012
Modeler,	0.1891	0.1904 (true, -averaged)	−0.0013
Critic, fully marginal	0.2107	—	—

On the risk-difference scale, the modeler’s additive-model estimates are not perfectly unbiased for the -specific truth — forcing additivity on and , when U’s marginalization technically induces a small interaction on the logit scale , leaves a residual bias of about 0.001 in each stratum. That residual is roughly fifteen to seventeen times smaller than the gap between the critic’s single marginal number (0.2107) and either patient type’s true absolute benefit (0.2311 for , 0.1904 for ). The critic’s number is not biased as an estimate of the population-average risk difference — it equals (0.2311+0.1904)/2 exactly, a consequence of linearity of expectation that holds regardless of the underlying model’s nonlinearity. It is, however, the wrong number for every actual patient, by an amount well over an order of magnitude larger than what the simple additive model adjusting only for leaves on the table.

Adding an interaction in the small model that did not exist in the full model would have made the -specific risk differences be exactly unbiased.

Missing Covariates and Mismodeled Measured Covariates

In Incorrect covariate adjustment may be more correct that adjusted marginal estimates I provide several relevant examples. First consider transportability of risk differences. In one example a marginal estimate computed as an 0.087 increase in risk among trial participants averaging age 70 becomes 0.130 in a simulated younger, more-female population — the same fitted model, the same treatment effect, but a materially different marginal number once the covariate distribution changes, because the marginal estimand is a function of whichever covariate values happened to enroll. Marginal estimands are sample-specific.

Next consider inference under an ill-fitting model. An example is presented in the above reference article which simulated treatment contrast’s Type I assertion probability for a misspecified linear-in-age, sex-omitted model came out essentially at or slightly below the nominal 0.05 level, and simulated power to detect a true OR of 1.75 was 0.812 for the misspecified model versus 0.813 for the correctly specified model — a negligible difference.

Summary

A general way of stating the fundamental clinical question was given that is completely concordant with parallel-group RCTs and with clinical decision making and does not involve unobservable quantities. Standard models, or even better, distribution-free semiparametric models, directly answer this fundamental question. This has several downstream ramifications, including the creating of the best framework for assessing differential treatment effects (interactions on a scale for which such effects are allowed to be zero when the treatment has an overall positive effect), handling partial information, and analyzing arbitrarily censored data and longitudinal data. Statistical models also form the bridge between frequentist and Bayesian approaches, the latter providing exact inference and providing a formal mechanism for incorporating external information. Statistical models are more robust than thought by critics, and their assumptions involve observable quantities so are capable of being critiqued. And models can be formulated to be fully concordant with the experimental design.

The Unifying Capabilities of Cumulative Probability Semiparametric Models

Frank Harrell — Sun, 26 Jul 2026 05:00:00 GMT

Disclaimer: No data were harmed during the making of this article. Any use of binning was strictly prohibited.

Background

Consider the oldest statistical regression model, the linear model, named from the fact that it is linear in the parameters. Suppose that the outcome variable is continuous, and conditional on a vector of baseline descriptors has a normal distribution. Let denote a vector of independent variables, a vector of regression coefficients corresponding to them, and denote their inner product, i.e., , also called the linear predictor. The linear model may be stated as where is a residual or error term having a normal distribution with mean zero. All regression models are flexible in allowing to be expanded to contain nonlinear and non-additive terms (interactions). The parametric linear model is inflexible with regard to the normality and constant variance assumptions for and the need for to already be properly transformed so that effects of the baseline variables are largely additive and the error term is also additive. Parametric methods are extremely sensitive to -transformations.

Now consider a very general statistical model that is part of the cumulative probability model (CPM) family of ordinal semiparametric models. The ordinal dependent variable may be continuous, discrete ordinal, count, a discrete/continuous mixture, or binary. Arbitrarily many ties in are handled automatically, as are floor and ceiling effects, lower and upper detection limits, and bi-modal distributions.

Discrete/continuous mixtures include continuous responses with clumping at zero and continuous variables with clinical event overrides at the extremes. Some values may be censored.

Continuous vs. Discrete Data Models

All continuous variables are also ordinal, although the reverse is not true. Much of statistical theory was developed for the setting where is truly continuous. Over a certain continuous domain it was assumed that all values could be measured. Continuous methods are frequently used as approximations for discrete where this is not the case. Some statististics decry such approximations, but the opposite situation is the most common: Most measurements have limited resolution and are discrete at the core. Some even have spikes in the distribution. Very few measurements made in research are truly continuous. Ordinal models better reflect reality and do not have to be considered as approximations. Ordinal models work just fine for continuous variables, as exemplified by the empirical distribution function, the nonparametric maximum likelihood estimator of the unknown cumulative distribution function. If the variable is discrete, the steps of the ECDF remain large as . Otherwise, the steps get progressively smaller. The ECDF works for all and all degrees of discreteness.

Let denote an increasing function that varies from 0 to 1 as varies from to . Standard cumulative distribution functions do this so are convenient choices for . A CPM may be stated as where is any of the observed distinct values of the outcome , means “given” or “conditional on”, and denotes a series of intercepts.

Most articles state CPMs in terms of but I use to make it consistent with how binary logistic models and Cox models are stated.

Note that no assumption is made that follows a distribution . is chosen because it is monotonically increasing and sometimes because it leads to interpretable effects on the linear predictor scale. Semiparametric models do not make absolute distributional assumptions, only relative ones. is the specific scale on which the distribution of is shifted by a shift in . More clearly, the model assumes that for two different values of , and , and are parallel to each other as a function of , where is the inverse function of , i.e., if (the expit function), , the logit function or logit link. This is called the proportionality or parallelism assumption. For the proportional hazards model, one assumes the are parallel for different covariate settings (here time takes the place of ). Proportionality/parallelism assumptions are relative distributional assumptions.

Parametric models have identical parallelism assumptions. In a linear model, parallelism comes from having the same variance of for different settings in addition to normality of holding for all . In addition, parametric models assume that the inverse (link) function of the cumulative probabilities is linear when plotted against . Think of examining linearity of a plot to check for normality of residuals in a parametric linear model. Linearity is a stronger assumption than parallelism.

See this for a gentle introduction to ordinal models.

In a CPM there are as many intercepts as there are distinct -values, less one. When the baseline variables have no effect, and the CPM is just a restatement of the empirical cumulative distribution function (ECDF), which is the nonparametric maximum likelihood estimator of the cumulative distribution of . Let’s work through that. The model in this case is and the nonparametric maximum likelihood estimates of the s is where is the link/inverse function of and is the empirical cumulative distribution function where is the indicator function: 1 if is true, 0 otherwise. is just .

Technical detail: We need to shift by one distinct value to get from the ECDF since it pertains to and we are modeling not .

The intercepts encode the empirical distribution of when or there is no . They will move around as needed to model any distribution of —bi-modal, discontinuous, etc. A spike in the distribution leads to a large jump in the intercept for that point. For the Cox proportional hazards model the intercepts represent the where is the underlying survival curve (a step function).

When , a CPM can be thought of as a shifted ECDF. CPMs are semiparametric because they are parametric with respect to and non-parametric with regard to the underlying unknown distribution. When has only two distinct levels and is the expit function, the CPM is the binary logistic model.

Let’s write the linear model in CPM form. This encapsulates all assumptions of the linear model in one formula. Let denote the cumulative distribution function of a standard normal distribution. is the ordinary -scaled intercept.

So and is absorbed in . The intercepts are linear in whereas they are arbitrarily descending in general semiparametric CPMs. Linear predictors in CPMs are unitless. This has an advantage of making CPMs for different comparable. For example

One can compare deviances for two fitted models using different link functions since the deviance under is link-independent. Deviances can be used to choose the best fitting link function.
One can compare standard errors of when is tragically dichotomized to the SE when is decimated into 10 bins to the SE when is kept continuous and not binned. Efficiency losses from using partial information can be computed empirically.

CPMs work perfectly well for continuous . The R orm function in the rms package has no limit on the number of intercepts. A sample of size with 10 covariates and no ties on (i.e., the model has 299,999 intercepts) can be fitted in 2.5 seconds.

This is possible because the part of the Hessian matrix corresponding to intercepts is tri-band diagonal, even if right-censoring is present. This is very sparse and leads to virtually instant matrix inversion.

Link Functions

The choice of , or equivalently its inverse, the link function , specifies the relative distributional assumption, i.e., the transformed scale for which CDFs are parallel. Commonly used distribution families/link functions are below with again denoting the standard normal CDF. The last column displays what connection the model assumes for the distributions of for different .

Table 1: Distribution families used in ordinal cumulative probability models. denotes the Gaussian cumulative distribution function. For the Connection column, . The connection specifies the only distributional assumption if the model is fitted semiparametrically, i.e, contains an intercept for every unique value less one. For parametric models, must be specified absolutely instead of just requiring a relationship between and . For example, the traditional Gaussian parametric model specifies that .

Distribution		Link Function	Link Name	Connection
Logistic			logit
Gaussian			probit
Gumbel maximum value
Gumbel minimum value			complementary
Cauchy			cauchit

The Gumbel maximum value distribution is also called the extreme value type I distribution. This distribution ( link) also represents a continuous time proportional hazards model.

General Advantages of CPMs

CPMs have the potential to revolutionize how applied statistics is taught. Instead of a hodge-podge of seemingly disconnected methods and the typical cookbook approach of teaching how to select from among them, a unified model with unified model diagnostics can be taught.
CPMs handle a huge variety of types of dependent variables and all types of censoring.
CPMs are robust, e.g., are unaffected by -outliers and and don’t change if is transformed by a monotonically increasing function.
They are efficient. For example the proportional odds model that compares two groups without covariate adjustment (i.e., the Wilcoxon test), is as efficient as the two-sample -test when normality and equal variances hold. When these do not hold, the semiparametric approach can be arbitrarily more efficient than the parametric approach.
They don’t assume anything about the distribution of for a given .
They encompass a huge number of standard methods under one model family.
CPMs provide a wide number of readouts including cumulative probabilities, survival curves, restricted mean survival time, probability of an outcome at a specific level , effect ratios, and covariate-specific differences in means and quantiles, along with uncertainty intervals. When doing a Wilcoxon test using the proportional odds model, the model can be used to estimate any measure of central tendency; nonparametric tests are thus no longer isolated from general estimation.
Being based on a standard likelihood function, penalized estimation can be straightforward.
Also because of being likelihood-based, CPMs are readily used in Bayesian modeling. For example, it is easy to obtain a Bayesian Wilcoxon test.

Special Cases

The CPM family of models encompasses an amazing list of statistical tests and models as special cases as summarized in the following table. CPM implementations that lead to the special cases are given, along with some links to further discussions. Note that the Wilcoxon signed-rank test, which is not transformation-invariant for , is now considered obsolete. It is replaced by the rank difference test of Kornbrot (1990) which is easily implemented in a proportional odds model by including subject random effects or using a robust cluster sandwich estimator to deal with having a pair of observations from each subject.

For some of the cases, a link to more information is given. is the link function, with “ = any” meaning the choice of link is irrelevant. The probit link is the normal CDF inverse function, the logit link is the inverse of the logistic CDF, and the link goes along with the Gumbel maximum value distribution used to generate the Cox proportional hazards model. below means that covariates are not included in the model.

Table 2: Special cases of CPMs

Test or Estimator	CPM Implementation	More Information
ECDF	=any,
Kaplan-Meier	=any, right-censored ,	ℹ️
Turnbull	=any, interval-censored ,	ℹ️
Proportions, sample mean, quantiles, pseudomedian	=any,
Pearson	binary, =any, indicator variables
Binary logistic	=logit, has 2 levels
Proportional odds model	=logit
Probit model	=probit
2-sample -test	=probit, binary,	ℹ️
paired -test	=probit, binary, random intercepts or sandwich estimator	ℹ️
-group ANOVA	=probit, has indicator variables
linear model	=probit,	ℹ️
2-sample Wilcoxon	=logit, binary	ℹ️
paired Wilcoxon	=logit, binary, random intercepts or sandwich estimator	ℹ️
-group Kruskal-Wallis	=logit, has indicator variables
Normal scores test	=probit, binary
logrank test	=log-log, right-censored , binary	ℹ️
Cox PH model	=log-log, censored	ℹ️
Accelerated failure time model	=probit or logit, censored	ℹ️

Note that accelerated failure time models often incorporate log-normal or log-logistic distributions which assume that the log transformation of time is a perfect transformation. The CPM approach provides the same estimates whether or not time is logged, so do not depend on the correctness of the log transformation.

Extensions for Longitudinal Data

CPM models readily extend to repeated measures/longitudinal binary, ordinal, and continuous data as discussed here and in Ordinal state transition models as a unifying risk prediction framework and here. Discrete time Markov Ordinal State Transition (MOST) models are the most general. One can instead use standard random intercepts if the within-subject correlation structure is compound symmetric (e.g., correlations do not decrease with increasing time gaps). A blend of the two is usually better: include a first-order Markov process and random intercepts. This allows for a longitudinal correlation structure that can range from a serial correlation AR(1) form to a compound symmetric exchangeable form.

The MOST approach unifies many other approaches, including the following as special cases:

Single follow-up time ordinal models
Cox proportional hazards model (transition probabilities are discrete hazards and odds ratios are almost identical to hazard ratios)
multi-state time-to-event models
recurrent events
recurrent events in the presence of an absorbing state

When subjects have similar numbers of longitudinal measurements and there are almost no missing measurements or dropouts, a GEE approach can be considered. A working independence correlation structure is often used to estimate the regression coefficients, then the standard errors are adjusted for intra-subject correlation using a cluster sandwich robust covariance estimator. This was done here so that a proportional odds model could provide a Wilcoxon-type rank difference test for paired data.

Summary

The CPM family of models handle huge varieties of types of dependent variables and of standard statistical tests and estimators. CPMs do this while maintaining excellent power and being more robust than distribution- or transformation-dependent parametric approaches. They provide a completely unified approach for the majority of common statistical analysis needs. Teaching of applied statistics should focus on CPMs.

Resources

Reuse

CC BY 4.0

Implications of the Draft FDA Bayesian Guidance

Frank Harrell — Thu, 21 May 2026 05:00:00 GMT

Events
- AstraZenica Global Statistical Forum and Washington Statistical Society 2026-05-21
Slides
Video

Thoughts About the Roles of AI for Statistics

Frank Harrell — Tue, 19 May 2026 05:00:00 GMT

Events
- Regression Modeling Strategies Course 2026 2026-05-19
Slides

Questions We Forget To Ask When Designing an RCT

Frank Harrell — Thu, 16 Apr 2026 05:00:00 GMT

Events
- JHU Trial Innovation Center DIDACT Symposium 2026-04-16
Slides

Causal by Design

Frank Harrell — Wed, 08 Apr 2026 05:00:00 GMT

Background

Two research methods are being used with increasing frequency: causal inference and target trial emulation. There are many complex situations where a formal causal calculus such as that developed by Judea Pearl is needed to allow one to infer than an effect is caused by a specific variable such as the use of a treatment or exposure to a specific agent. Practitioners of causal calculus propose that it to be a necessary ingredient of virtually every causal inference including that from tightly controlled randomized studies.

Practitioners of target trial emulation claim TTE to be a valuable component of observational treatment comparisons, even though TTE per se does not address the all-important issue of confounding, especially confounding by indication, where one treatment is selected over another due to prognostic characteristics such as disease severity, drug tolerability, or family income. TTE instills good thinking about the inception cohort, time zero, and time-dependent covariates, but does not emulate randomization, blinding of measurements to treatments, or provide any additional help with confounding. The emulation in TTE is a misnomer. I wish that TTE was called BPOC (Best Practices in Observational Comparisons).

TTE is already being used in a haphazard fashion in articles in medical journals, with authors not bothering even to address the potential for confounding and some authors being so naive as to merely state “we assume that all confounders were part of data collection”. Observational treatment comparisons would certainly be simple were that the case!

Study Design

Before delving more into issues introduced above, let’s consider, for our purposes, four types of studies.

Randomized controlled experiments in which there is no interruption of the flow in the data generation process and there are no auxiliary time-dependent variables being considered.
Randomized experiments in which there is an interruption in the flow, or one is interested in answering a “what if” question related to a time-dependent (post-randomization) covariate.
Prospective observational studies with very intentional protocolized collection of high quality data.
Non-prospective, non-protocolized observational studies, i.e., undesigned studies.

In design 1, the response variable is measured on every subject and there are no “what if” questions such as “what would the difference in response between the drug groups have been had all of the animals in the experiment fully metabolized the new drug?”. An example of design 2 is a randomized trial of an anti-hypertension drug taking drug adherence into account. The usual as-randomized intent-to-treat (ITT) analysis contrasts subjects assigned to take drug A with those assigned to take drug B, so the inference pertains to what happens when non-adherence to the assigned treatment is ignored and averaged out. A non-ITT design 2 analysis may ask “but what is the efficacy if everyone adhered to the drug?”. To infer that faithfully taking the drug causes a larger reduction in blood pressure requires complex thinking.

Design 3 is the only observational study design for which one can hope to make a causal inference about an exposure effect. In this design, one addresses confounding well in advance of starting data collection. This is done by assembling a good many experts in the decision process used to select therapies in the field. The list of possible selection factors is elicited from the experts, and data collection is designed to ensure accurate collection of these variables, with minimal missingness. The need for observational studies to be actually designed is not even mentioned in some review articles about TTE. Investigators should never be given carte blanche just because they desire to save money and time by using routinely collected data.

Design 4 is hopeless except in very simple situations that are amazingly well understood and where one can have confidence in the quality and unbiasedness of the data.

Implications of Study Design on Causal Inference

cause: something without which something else would not happen — Cambridge American Dictionary

In design 1 above, there is no explanation for the result other than the experimental manipulation and random variation. Statistical analysis takes care of quantifying uncertainty in estimated treatment effects taking random variation into account. One can even use a randomization test to almost nonparametrically test for treatment effect. Causal calculus is elegant and some researchers may prefer to include a DAG in the paper, but this is not strictly necessary for the primary analysis. The process of elimination excludes all other mechanisms other than irreducible noise in the system. One can create a DAG for each experimental design and use the same DAG for all implementations of that design, across many design 1 studies.

I don’t recommend randomization tests because they don’t provide all the estimates and uncertainty intervals we need, and don’t extend to more complex situations such as handling random effects and longitudinal data. They also don’t admit Bayesian analysis which handles many other facets of the research, including incorporation of high-quality outside information and provision of an elegant sequential inferential framework.

Design 2 is the perfect spot for causal inference. Asking “what if” questions such as “what if everyone adhered to their assigned drug” are perfect problems for application of causal calculus. The simplest causal inference, which applies to a subset of the problems where patients either always adhere or never adhere to the assigned drug, uses instrumental variables. An instrumental variable must affect the dependent variable only through the treatment, have no direct effect on the outcome itself, and be independent of unmeasured confounders. The randomized treatment assignment is the perfect instrument.

Observational designs 3 and 4 are in great need of formal causal inference. Design 4 usually remains hopeless no matter how elegant the causal calculus that is used. Even a higher-cost design 3 can fail when the consulted experts fail to list an important confounder that was not included in data collection.

The only hope for design 4 requires all of the following steps to be faithfully executed when conducting an observational treatment comparison:

Involve 10 or more true experts in medical care and medical decision making, eliciting from them the list of potential factors used in medical care decisions. Make certain that the experts have no knowledge of which variables are collected, so that they do not engage in data availability bias.
Pool all the experts’ variables and make sure they are present in the dataset to be used, have minimal missingness, and are reliably collected.
Audit 200 or more patients and verify the accuracy of all the variables being analyzed.
Use TTE principles.

Summary

Causal calculus such as the system developed by Pearl is elegant and can provide a great deal of insight for complex research questions. But straight-up randomized experiments are causal by design. Observational studies should be prospectively designed so that causal inference has a chance. Target trial emulation runs the risk of serving as window dressing for inadequately designed observational studies. Any method that claims to emulate what RCTs do must address

prospective protocols for study conduct, patient recruitment, data acquisition, patient care, treatment administration, concomitant therapies, safety monitoring, etc.
random allocation
blinding (including blinded evaluation of patient responses)
blocking and other handling of uncontrolled variables
sources of outcome variation
measurement processes and their standardization
concurrent controls in time and space
reasons patients enter and leave the cohort
interventional consistency
pre-specification of statistical analysis plans
consistent outcome evaluation
uniformity of follow-up across treatment groups
recall bias
solid time 0 definition

Reuse

CC BY 4.0

Goal-Driven Flexible Bayesian Design

Frank Harrell — Wed, 17 Sep 2025 05:00:00 GMT

Events
- Vanderbilt Department of Biostatistics Seminar, Nashville TN USA 2025-09-17
- ACTStats 2025 Annual Meeting Keynote Talk, Nashville TN USA
Video; see also this
Slides
Details

Measures of Central Tendency for an Asymmetric Distribution, and Confidence Intervals

Frank Harrell — Thu, 14 Aug 2025 05:00:00 GMT

Measures of Central Tendency

For symmetric normal-like distributions there is a clear winner for measuring central tendency: the sample mean. The mean has the highest precision/efficiency and is also representative of a typical observation from the population distribution. The mean is not robust, e.g., is too affected by extreme values, when the distribution is heavy-tailed or asymmetric. For general continuous distributions with arbitrarily heavy tails or skewness, the sample median is robust and representative of a typical observation. But with a normal distribution the median is imprecise (higher variance, wider confidence limits) and has efficiency of only compared to the mean. A sample of size 100 properly analyzed using a mean will revert to an effective sample size of 64 when using the median if normality holds.

The sample mean has a breakdown point of 0, i.e., can be too affected by a single contaminated observation. The sample median has a breakdown point of 0.5, meaning it can withstand up to half of the observations being contaminated.

For small samples the Harrell-Davis smooth quantile estimator improves precision in estimating population quantiles. For the median, the HD estimate is the average sample median over infinitely many bootstrap resamples, although the algorithm is quite fast. It is implemented in the Hmisc package hdquantile function, which estimates the standard error of the estimate using the leave-out-one jackknife.

The pseudomedian (see also here), also called the Hodges-Lehmann estimator, is less well known. It is affiliated with the Wilcoxon signed-rank test and can be computed by inverting that test, i.e., by finding the value that minimizes the signed rank statistic.. It is computed by taking all possible pairs of observations (including pairing an observation with itself), computing the midpoint of each pair, and computing the median of the midpoints. The R Hmisc package’s pMedian function can compute the pseudomedian on 250,000 observations in 0.09 seconds.

The pseudomedian has a breakdown point of 0.29

When the distribution is normal, the efficiency of the pseudomedian compared to the mean is . When the distribution is symmetric, the pseudomedian is often much better than the sample mean or median in estimating the population mean and median.

Symmetry implies that the mean, median, and pseudomedian all estimate the same quantity, which is both the mean and the median since they are identical.

Let’s see how often the pseudomedian is between the mean and median when sampling from a normal distribution of size and when the estimates non-trivially differ. Repeat for a lognormal distribution. Frequencies of NAs pertain to the number of samples (out of 5000) for which at least two of the three estimates were trivially different.

require(Hmisc)
require(data.table)
require(boot)
require(bayesboot)
require(ggplot2)
options(datatable.print.class = FALSE)
g <- function(lognorm=FALSE) {
  x <- rnorm(50)
  if(lognorm) x <- exp(x)
  m    <- mean(x)
  med  <- median(x)
  pmed <- pMedian(x)
  mindif <- if(lognorm) 0.2 else 0.02
  dif <- min(diff(sort(c(m, med, pmed))))
  if(dif > mindif)
    (m < pmed & pmed < med) | (m > pmed & pmed > med)
  else NA
}
set.seed(1)
r <- replicate(5000, g())
table(r, useNA='ifany')

r
FALSE  TRUE   
  290  1638  3072

r <- replicate(5000, g(lognorm=TRUE))
table(r, useNA='ifany')

r
TRUE  
2299 2701

Extension of the Pseudomedian to Difference Between Groups

When summarizing the difference between two independent groups of observations, we do not take the difference in pseudomedians, but instead use a more direct two-sample Hodges-Lehmann estimator: the median of all pairwise differences between an observation in group B and an observation in group A. This is the estimator that is zero when the Wilcoxon-Mann-Whitney two-sample test statistic is at its most null value.

It is always more difficult to estimate a difference than it is to estimate a one-sample quantity. Recall that in the normal case the variance of a difference in means equals the sum of the two variances. When there are observations in each of the two groups, the variance a mean of observations is and the variance of the mean difference is . For the pseudomedian on observations we compute the median of midpoints. For differences in the balanced case we compute the median of differences. The ratio of the number of one-sample pairs to the number of two-sample pairs is instead of the 4 in the parametric case. This takes into account that for the pseudomedian every pair shares an observation with other pairs.

Confidence Intervals

As discussed here, there is no nonparametric confidence interval for the mean of a general distribution. The mean is inherently a “parametric” quantity, unlike the sample median. Most data analysts mistakenly rely by default on the central limit theorem (CLT) to compute confidence limits for a population mean. A key problem arises: the population variance is seldom known. So analysts substitute the sample variance and proceed as if the variance is known, not aware they may be violating a key requirement of the CLT: the sample mean and sample variance must be independent in order for the standardized mean to converge with any rapidity to the normal distribution. For symmetric distributions this is OK, but not for asymmetric ones. Underlying this issue is that in practice one does not know how to transform the data to achieve normality, and we desire to estimate the population mean on the original scale. If the distribution were known to be lognormal, the analyst would analyze the data after taking logs. But there is no oracle to lead the analyst to this action, and if you should have taken logs but didn’t, problems arise. Even if you knew to take logs, it is still a difficult problem to get confidence intervals for the population mean on the original scale (which is why the smearing estimator exists), although the formula for the population mean itself is a simple function of on the log scale.

Proceeding without an oracle nearby, let be the 0.975 quantile of the normal distribution (1.96), and denote the sample standard deviation of the data values. Standard use of the CLT yields the following 0.95 confidence limits: . This formula is quite accurate even for as low as 20 when the data come from a normal distribution.

In all that follows we consider samples from the lognormal distribution with on the log scale. As shown in Biostatistics for Biomedical Research, the above CLT-based formula leads to very inaccurate confidence intervals. Even for the non-coverage in the two tails is 0.0182 on the left and 0.0406 on the right when both were supposed to be 0.025 for an overall 0.95 CI. Coverage is much worse for small . Replacing with the critical value from a distribution will not help, as the problem is with asymmetry and not just heavy tails.

There are a number of ad hoc corrections to the CLT formula we will try. First, in recognition of asymmetry let’s use dual SDs instead of a single sample SD. That is, compute by taking the observations that are below the sample mean and computing their root mean squared average distance from the overall mean (we actually divide the sum of squares by one less than the number of low observations). Do likewise to compute , the upper SD. Take the confidence limits to be and , where is the overall sample size. Variations of this will also be tried, where instead of splitting on the mean we split on the median and on the pseudomedian.

Other ad hoc CI substitutes are tried below:

Substitute Gini’s mean difference for a single SD, scaling it by a factor that makes it estimate the SD unbiasedly if the distribution were normal
Shift a confidence interval for the pseudomedian by the difference between the mean and pseudomedian
Determine which quantiles of a standard normal distribution correspond to and use those empirical quantiles from the raw data as the CI
The same but use the Harrell-Davis quantile estimator
Use 0.025 and 0.975 quantiles of the data shrunk towards the mean by a factor of .

But we expect that bootstrap confidence intervals will be more accurate, not to mention having better theory supporting them. . The problem is that there are many bootstrap variants. Based on theory and experience, the bias-corrected accelerated method called BCa is expected to perform best, because it explicitly handles asymmetry in sampling distributions. We can also add our own variations to the common bootstrap variants:

Charles Geyer has excellent presentations about the bootstrap and limitations of some of the bootstrap variants. See this and this

When using normal approximation bootstraps based on bootstrap SDs of the estimates, substitute dual SDs as above
Substitute the Harrell-Davis quantile estimator for nonparametric percentile bootstrap intervals

When using the sample median as the central tendency measure, there is for once an exact nonparametric confidence interval. The R function below computes it. This was taken from the DescTools package SignTest.default function. See also this and this.

cimed <- function(x, alpha=0.05, na.rm=FALSE) {
  if(na.rm) x <- x[! is.na(x)]
  n <- length(x)
  k <- qbinom(p=alpha / 2, size=n, prob=0.5, lower.tail=TRUE)
  # Actual CL coverage: 1 - 2 * pbinom(k - 1, size=n, prob=0.5) >= 1 - alpha
  sort(x)[c(k, n - k + 1)]
  }

For the Harrell-Davis median estimator we assume normality and use the jackknife standard error to compute confidence limits.

Not studied here is empirical likelihood, which was a promising approach when it was developed by Owen in 1990. But I tried it for the situation studied here and confidence coverage for the mean was terrible. Advocates of empirical likelihood suggested putting a bootstrap loop around the already computationally intensive combinatorial empirical likelihood algorithm, in order to get accurate CIs. But why go to that trouble instead of just using the bootstrap, or why not use the double bootstrap to get highly accurate intervals with much less computation than empirical likelihood?

The double bootstrap is used to correct a single bootstrap. The most efficient double bootstrap algorithm available increases computation time by over a single bootstrap.

Update: Various other boostrap variants were tried including a triple bootstrap, with unsatisfactory results. See a summary at the end of this article. This summary also better explains the fundamental limitations of mean-based methods for small skewed samples.

Distribution and True Location Values

As mentioned above, the distribution from which we’ll simulate samples is the lognormal distribution with .

mu         <- 0
sigma      <- 1.65
truemean   <- exp(sigma ^ 2 / 2)
truemedian <- exp(mu)

This is a highly skewed distribution with median and mean 3.9011. There is no analytic expression available for the pseudomedian, so we compute it by simulating a sample of size .

set.seed(1)
truepmedian <- pMedian(rlnorm(250000, mu, sigma))
truepmedian

[1] 1.571895

truevals <- c(mean = truemean, median = truemedian, pseudomedian = truepmedian)

The probability density function is shown below, with vertical lines depicting the three location measures (blue:mean, red:median, gray:pseudomedian).

par(mar=c(3, 3, 1, 1), mgp=c(2, .43, 0))
curve(dlnorm(x, mu, sigma), 0, 8, ylab='Probability Density')
abline(v=truemean,    col='blue')
abline(v=truemedian,  col='red')
abline(v=truepmedian, col='gray')

Distributions of Sample Location Estimates

Let’s see how much asymmetry exists in the distribution of sample estimates of central tendency by computing sample estimates from each of 5000 samples from our chosen log-normal distribution. Population values are shown with red vertical lines. Symmetry is quantified by the ratio of the SD of those values above the grand mean to the SD of those below.

set.seed(1)
N <- 200
g <- function() {
  x <- rlnorm(N, mu, sigma)
  c(mean=mean(x), median=median(x), hdmedian=unname(hdquantile(x, 0.5)), pmedian=pMedian(x))
}
r <- t(replicate(5000, g()))
sym <- function(x) {
  s <- dualSD(x)
  ratio <- s['top'] / s['bottom']
  title(sub=paste('Symmetry:', round(ratio, 2)), line=-8, adj=1)
}

mmap <- .q(mean=mean, median=median, hdmedian=median, pmedian=pseudomedian)

par(mfrow=c(2,2))
for(p in colnames(r)) {
  u <- r[, p]
  hist(u, nclass=100, xlab=p, main=NULL); sym(u)
  abline(v=truevals[mmap[p]], col='red')
}

There is strong asymmetry in the sample means, which needs to be taken into account when constructing confidence limits which by necessity must also be asymmetric. Medians and pseudomedians have much more symmetric distributions.

Out of curiosity, compare sample medians with Harrell-Davis estimates.

ggplot(r, aes(x=abs(median - hdmedian))) + geom_histogram(bins=50)

Also compute the root mean squared error for the ordinary sample median vs. the HD median.

rmse <- function(x) sqrt(mean((x - truemedian) ^ 2))
rmse(r[, 'median'  ])

[1] 0.1466573

rmse(r[, 'hdmedian'])

[1] 0.1442742

We see that for the HD median was not worth the trouble. HD may have offered more precision for outer quantiles.

CIs Computed for One Sample

Generate a sample of from the chosen log-normal distribution and compute all the confidence limits we will study.

w <- 'mean,CLT,usual
mean,CLT,dual SD pm
mean,CLT,dual SD med
mean,CLT,dual SD mean
mean,CLT,Gini
mean,boot,normal
mean,boot,basic
mean,boot,student
mean,boot,percentile
mean,boot,BCa
mean,boot,normal dual SD pm
mean,boot,normal dual SD med
mean,boot,normal dual SD mean
mean,boot,percentile HD
mean,boot,Bayesian
mean,shifted pm,
mean,CLT,percentile HD
mean,shrunk HD,
median,exact,
hdmedian,CLT,usual
pseudomedian,exact,
pseudomedian,boot,BCa
pseudomedian,boot,normal
pseudomedian,boot,percentile
pseudomedian,boot,basic'
what <- fread(text=w, col.names=c('measure', 'method', 'variant'), header=FALSE, fill=TRUE)
what[, type := 1 : nrow(what)]

cis <- function(x, exact=FALSE) {
  z     <- qnorm(0.975)
  m     <- mean(x)
  med   <- median(x)
  hdmed <- unname(hdquantile(x, 0.5, se=TRUE))
  pmed  <- pMedian(x)
  n     <- length(x)
  est   <- c(mean=m, median=med, hdmedian=hdmed, pseudomedian=pmed)
  estimate <- unname(what[, est[measure]])

  u <- z / sqrt(n)

  # nmin=10000 means use pooled SD if n < 100000
  dsd <- function(x, j) {
    dualSD(x, nmin=if(j == 1) 100000 else 2,
           center=switch(j, NA, pMedian(x), median(x), m ))
  }

  # Do all bootstrap resamples of the mean
  stat <- function(x, i) {
    x <- x[i]
    m <- mean(x)
    s <- sd(x)
    se <- s / sqrt(length(x))
    c(m, se)
  }
  b   <- boot(x, statistic=stat, R=1000)
  bmeans <- b$t[, 1]
  bci    <- boot.ci(b)[c('normal', 'basic', 'student', 'percent', 'bca')]
  lcl    <- function(z) z[length(z) - 1]
  ucl    <- function(z) z[length(z)]

  pmedb <- function(x, i) pMedian(x[i])
  # Do all bootstrap resamples of the pseudomedian
  bpm   <- boot(x, statistic=pmedb, R=1000)
  bpmci <- boot.ci(bpm, type=c('norm', 'perc', 'bca', 'basic'))

  # Create a list() of list()'s of computed lower and upper confidence limits so
  # that we can easily process all of them with sapply()
  res <- list(
    {
      # CLT
      lo <- hi <- numeric(4)
      for(j in 1:4) {
        s <- dsd(x, j)
        lo[j] <- m - u * s['bottom']
        hi[j] <- m + u * s['top']
      }
      list(lo=lo, hi=hi)
    },
    {
      # CLT using Gini's mean difference instead of SD
      s <- GiniMd(x) * sqrt(pi / 4)
      list(lo = m - u * s, hi = m + u * s)
    },
    list(lo = sapply(bci, lcl), hi = sapply(bci, ucl)),
    {
      # Bootstrap normal approximation CIs using dual SDs
      lo <- hi <- numeric(3)
      for(j in 2 : 4) {
        s <- dsd(bmeans, j)
        lo[j - 1] <- m - z * s['bottom']
        hi[j - 1] <- m + z * s['top']
      }
      list(lo=lo, hi=hi)
    },
    {
      # Bootstrap percentile CI using Harrell-Davis quantile estimator
      p <- hdquantile(bmeans, c(0.025, 0.975), names=FALSE)
      list(lo = p[1], hi=p[2])
    },
    {
      # Bayesian bootstrap
      wmean <- function(x, wts) sum(wts * x) / sum(wts)
      bb <- bayesboot(x, statistic=wmean, use.weights=TRUE, R=1000)
      s  <- summary(bb)
      list(lo=s$value[3], hi=s$value[4])
    },
    {
      # Compute shifted Hodges-Lehmann CI (shifted pseudomedian)
      w <- wilcox.test(x, conf.int=TRUE, exact=exact)
      shift <- m - w$estimate
      list(lo=w$conf.int[1] + shift, hi=w$conf.int[2] + shift)
    },
    {
      # CLT HD percentile
      p  <- pnorm(u)
      qu <- hdquantile(x, c(1 - p, 0.5, p))
      list(lo=m - qu[2] + qu[1], hi=m - qu[2] + qu[3])
    },
    {
      # Mean, shrunk Harrell-Davis 0.025 0..975 quantiles
      qu <- hdquantile(x, c(0.025, 0.975))
      w1 <- (m - qu[1]) / sqrt(n)
      w2 <- (qu[2] - m) / sqrt(n)
      list(lo = m - w1, hi=m + w2)
    },
    {
      # Exact nonparametric CI for median
      qu <- cimed(x)
      list(lo=qu[1], hi=qu[2])
    },
    {
      # HD approximate CI for HD median
      se <- attr(hdmed, 'se')
      list(lo=hdmed - z * se, hi=hdmed + z * se)
    },
    {
      # Exact nonparametric CI for pseudomedian under symmetry assumption
      w <- wilcox.test(x, conf.int=TRUE, exact=TRUE)
      list(lo=w$conf.int[1], hi=w$conf.int[2])
    },
    {
      # Bootstrap BCa confidence interval for the pseudomedian
      list(lo=lcl(bpmci$bca), hi=ucl(bpmci$bca))
    },
    {
      # Bootstrap normal approx for pseudomedian
      list(lo=lcl(bpmci$normal), hi=ucl(bpmci$normal))
    },
    {
      # Bootstrap percentile for pseudomedian
      list(lo=lcl(bpmci$percent), hi=ucl(bpmci$percent))
    },
    {
      # Basic bootstrap for pseudomedian
      list(lo=lcl(bpmci$basic), hi=ucl(bpmci$basic))
    }
    )

  lower <- unlist(sapply(res, function(x) x$lo))
  upper <- unlist(sapply(res, function(x) x$hi))

  data.table(type = 1:length(lower), estimate=estimate, Lower=lower, Upper=upper)
}

set.seed(2)
x <- rlnorm(N, mu, sigma)
cis(x)

    type  estimate     Lower    Upper
 1:    1 3.9084646 2.9241099 4.892819
 2:    2 3.9084646 3.4288988 5.355652
 3:    3 3.9084646 3.4074705 5.211039
 4:    4 3.9084646 3.4750995 5.798228
 5:    5 3.9084646 3.2059036 4.611026
 6:    6 3.9084646 2.8833772 4.903142
 7:    7 3.9084646 2.8899896 4.900889
 8:    8 3.9084646 2.9595756 5.034969
 9:    9 3.9084646 2.9160400 4.926940
10:   10 3.9084646 2.9544121 4.985171
11:   11 3.9084646 2.9133207 4.934543
12:   12 3.9084646 2.9024144 4.923173
13:   13 3.9084646 2.8983605 4.919133
14:   14 3.9084646 2.9144900 4.925161
15:   15 3.9084646 2.9558150 4.879749
16:   16 3.9084646 3.4308936 4.540176
17:   17 3.9084646 3.6274924 4.341686
18:   18 3.9084646 3.6346241 5.674496
19:   19 0.9616767 0.6257150 1.612685
20:   20 0.9749037 0.5449554 1.404852
21:   21 1.8172888 1.3398304 2.447722
22:   22 1.8172888 1.3160193 2.475189
23:   23 1.8172888 1.2684055 2.357769
24:   24 1.8172888 1.2965646 2.443881
25:   25 1.8172888 1.1906965 2.338013
    type  estimate     Lower    Upper

Simulation of CI Coverage

As an aside, many statisticians make the mistake of simulating only the overall confidence coverage to check against nominal levels such as 0.95. This can hide significant coverage errors in both tails, just because the two non-coverage probabilities may luckily sum to around 0.05. We will separately estimate non-coverage in each tail.

Run 5000 experiments using our chosen lognormal distribution.

set.seed(3)
g <- function() {
  x <- rlnorm(N, mu, sigma)
  cis(x)
}
# Run reps simulations on one core
run1 <- function(reps, showprogress, core) {
  d <- data.table(sim = 1 : reps)
  r <- d[, g(), by=.(sim)]
  r[, sim := paste(core, sim, sep='-')]
  r
  }

if(file.exists('sim.rds')) R <- readRDS('sim.rds')  else {
  # The following tool 2 min. using 11 cores
  R <- runParallel(run1, reps=5000, seed=3)   # see https://hbiostat.org/rflow/parallel
  saveRDS(R, 'sim.rds')
}

Show distribution of confidence limits by method, separating means from other location measures. The spike histograms are interactive. Hover over elements to see more information.

r <- copy(R)
r <- r[what, on=.(type)]
r[, z := paste(method, variant)]
curtail <- function(x, a, b) pmin(pmax(x, a), b)
s <- r[measure == 'mean',]
s[, Lower := curtail(Lower, -1, 9)]
s[, Upper := curtail(Upper, 2.5, 15)]
s[, histboxp(x=Lower, group=z, bins=200)]

s[, histboxp(x=Upper, group=z, bins=200)]

s <- r[measure != 'mean',]
s[, histboxp(x=Lower, group=paste(measure, z), bins=200)]

s[, histboxp(x=Upper, group=paste(measure, z), bins=200)]

# For a different look try:
# ggplot(R, aes(Lower, Upper)) + geom_bin2d(bins=150) +
#   viridis::scale_fill_viridis(trans='log10', option='inferno') + facet_wrap(~ type)

Check true values against means over simulations.

mmap <- function(a) ifelse(a == 'hdmedian', 'median', a)
r[, trueval := truevals[mmap(measure)]]
r[, .(true = mean(trueval), mean_estimate = mean(estimate)), by=measure]

        measure     true mean_estimate
1:         mean 3.901067      3.882731
2:       median 1.000000      1.013085
3:     hdmedian 1.000000      1.024032
4: pseudomedian 1.571895      1.589268

Compute non-coverage proportion for each tail for each method.

error <- r[, .(low  = mean(Lower > trueval),
               high = mean(Upper < trueval)), by=.(measure, method, variant)]

Compute the total absolute error in the two tails.

error[, total_error := abs(low - 0.025) + abs(high - 0.025)]
error

         measure     method             variant    low   high total_error
 1:         mean        CLT               usual 0.0008 0.1428      0.1420
 2:         mean        CLT          dual SD pm 0.2090 0.0602      0.2192
 3:         mean        CLT         dual SD med 0.2014 0.0826      0.2340
 4:         mean        CLT        dual SD mean 0.2284 0.0314      0.2098
 5:         mean        CLT                Gini 0.1288 0.2844      0.3632
 6:         mean       boot              normal 0.0010 0.1428      0.1418
 7:         mean       boot               basic 0.0006 0.1674      0.1668
 8:         mean       boot             student 0.0024 0.1114      0.1090
 9:         mean       boot          percentile 0.0040 0.1248      0.1208
10:         mean       boot                 BCa 0.0220 0.0864      0.0644
11:         mean       boot   normal dual SD pm 0.0024 0.1302      0.1278
12:         mean       boot  normal dual SD med 0.0020 0.1322      0.1302
13:         mean       boot normal dual SD mean 0.0030 0.1274      0.1244
14:         mean       boot       percentile HD 0.0038 0.1260      0.1222
15:         mean       boot            Bayesian 0.0060 0.1394      0.1334
16:         mean shifted pm                     0.2602 0.3582      0.5684
17:         mean        CLT       percentile HD 0.3224 0.4526      0.7250
18:         mean  shrunk HD                     0.2884 0.0592      0.2976
19:       median      exact                     0.0232 0.0192      0.0076
20:     hdmedian        CLT               usual 0.0278 0.0438      0.0216
21: pseudomedian      exact                     0.0292 0.0364      0.0156
22: pseudomedian       boot                 BCa 0.0262 0.0286      0.0048
23: pseudomedian       boot              normal 0.0138 0.0494      0.0356
24: pseudomedian       boot          percentile 0.0228 0.0306      0.0078
25: pseudomedian       boot               basic 0.0096 0.0728      0.0632
         measure     method             variant    low   high total_error

Compute the mean CI widths for the methods when estimating the mean. Print them in order of total coverage error.

w <- r[measure == 'mean', .(length = mean(Upper - Lower)), by=.(measure, method, variant)]
u <- error[w, on=.(measure, method, variant)]
i <- u[, order(total_error)]
u[i, ]

    measure     method             variant    low   high total_error    length
 1:    mean       boot                 BCa 0.0220 0.0864      0.0644 3.6604757
 2:    mean       boot             student 0.0024 0.1114      0.1090 3.5307561
 3:    mean       boot          percentile 0.0040 0.1248      0.1208 3.0312429
 4:    mean       boot       percentile HD 0.0038 0.1260      0.1222 3.0334265
 5:    mean       boot normal dual SD mean 0.0030 0.1274      0.1244 3.1013843
 6:    mean       boot   normal dual SD pm 0.0024 0.1302      0.1278 3.0890244
 7:    mean       boot  normal dual SD med 0.0020 0.1322      0.1302 3.0829226
 8:    mean       boot            Bayesian 0.0060 0.1394      0.1334 2.8555153
 9:    mean       boot              normal 0.0010 0.1428      0.1418 3.0909886
10:    mean        CLT               usual 0.0008 0.1428      0.1420 3.1022174
11:    mean       boot               basic 0.0006 0.1674      0.1668 3.0312429
12:    mean        CLT        dual SD mean 0.2284 0.0314      0.2098 3.9299538
13:    mean        CLT          dual SD pm 0.2090 0.0602      0.2192 2.8775268
14:    mean        CLT         dual SD med 0.2014 0.0826      0.2340 2.6264842
15:    mean  shrunk HD                     0.2884 0.0592      0.2976 2.1110165
16:    mean        CLT                Gini 0.1288 0.2844      0.3632 1.4404794
17:    mean shifted pm                     0.2602 0.3582      0.5684 0.8266577
18:    mean        CLT       percentile HD 0.3224 0.4526      0.7250 0.4735200

# To display the table graphically use
# ggplot(u, aes(total_error, length, color=type)) + geom_point()

The overall winner for CI for the mean is the BCa bootstrap, which has tail non-coverage probabilities of 0.022 and 0.086 for overall coverage of 0.89 against the target of 0.95. This is not completely satisfactory, and a better confidence interval method may exist. If one wanted to trade off some coverage accuracy to get tighter CIs, the bootstrap percentile method might be considered.

As shown here, BCa yields poor coverage when . For and BCa again performed the best, but with 0.014 non-coverage on the left and 0.154 on the right. Even for , no bootstrap variant got extremely close to 0.025 non-coverage in both tails, but BCa was still the best.

Ad hoc methods of computing confidence intervals for the mean had embarrassing performance. The usual CLT formula was unsatisfactory in both tails, and attempted fixes made things worse.

For the median, the exact nonparametric method works extremely well as advertised. The method based on the HD estimate and its standard error are satisfactory.

The real surprise comes from the performance of confidence intervals for the population pseudomedian. Both the classic exact Hodges-Lehmann confidence interval (which assumes symmetry of the population distribution) and the BCa bootstrap method were excellent, with BCa being almost perfect. The exact method probably works as well as it does because the sampling distribution of the pseudomedian for was shown above to be very symmetric. The R wilcox.test will compute exact confidence limits for the pseudomedian quickly if , and approximate confidence limits quickly if . Because BCa worked so well, and because the pMedian function is so fast for up to 250,000 (unlike the wilcox.test function), BCa would be a good default method. However BCa is slow for , and a good replacement for it is the bootstrap nonparametric percentile CI in such cases. Beginning with Hmisc version 5.2-4, the pMedian function optionally efficiently computes bootstrap percentile (the default for ) or BCa (the default for ) confidence intervals.

Timing and Agreement of Percentile and BCa Confidence Limits for Pseudomedian

For varying sample sizes let’s check agreement and execution time of two types of confidence limits for the pseudomedian. One thousand bootstrap resamples are used throughout.

d   <- expand.grid(n=10:500, type=c('percentile', 'bca'))
setDT(d)
g <- function(n, type) {
  set.seed(n)   # make percentle and bca use the same data for each n
  y   <- rlnorm(n, mu, sigma)
  t1  <- system.time(a <- pMedian(y, conf.int=0.95, type=as.character(type)))['elapsed']
  list(time=t1, lower=a['lower'], upper=a['upper'])
}
if(file.exists('perc_vs_bca.rds')) u <- readRDS('perc_vs_bca.rds') else {
  u <- d[, g(n, type), by=.(n, type)]
  saveRDS(u, 'perc_vs_bca.rds')
}
ggplot(u, aes(x=n, y=time, color=type)) + geom_smooth(se=FALSE) +
  ylab('Execution Time, sec.')

Execution time is noticeably longer for BCa for .

lo <- u[, lower]
hi <- u[, upper]
lim <- function(x) ylim(quantile(x, c(0.02, 0.98)))
ggplot(u, aes(x=n, y=lower, color=type)) + geom_smooth(se=FALSE) +
  geom_point(size=0.2, alpha=0.5) + lim(lo)

ggplot(u, aes(x=n, y=upper, color=type)) + geom_smooth(se=FALSE) +
  geom_point(size=0.2, alpha=0.5) + lim(hi)

Lower confidence limits for BCa and the percentile bootstrap are almost indistinguishable on average, especially for . Upper limits converge around . This informs the default for the Hmisc pMedian function: BCa for and percentile otherwise.

Instead of summarizing averages as was done above, look at individual sample differences between the two CI methods.

w <- dcast(u, n ~ type, value.var=c('lower', 'upper'))
ggplot(w, aes(x=n, y=abs(lower_percentile - lower_bca))) + geom_smooth(se=FALSE) +
  geom_point(size=0.2, alpha=0.5) + ylim(0, 0.1)

ggplot(w, aes(x=n, y=abs(upper_percentile - upper_bca))) + geom_smooth(se=FALSE) +
  geom_point(size=0.2, alpha=0.5) + ylim(0, 0.3)

Width of “Exact” Confidence Intervals and Planning Studies for Precision

Compute the width of “exact” (under an assumption) CIs. Let the population pseudomedian be and assume that sample pseudomedians are from a location family such that they have a distribution that does not depend on (the population pseudomedian) but is then shifted by . Let the zero-shift (underlying / standardized) sampling distribution be . Then the exact 0.025 and 0.975 confidence limits of the pseudomedian are where is the sample pseudomedian, is the 0.025 quantile of , and is the 0.975 quantile of . The width of the exact CI in this case is simply the difference in these outer quantiles, although the CI center moves with sample pseudomedians. Compute the quantiles and hence the width as a function of n by simulation.

set.seed(1)
g <- function(n) pMedian(rlnorm(n, mu, sigma)) - truepmedian
tlim <- function(n) {
  pm <- t(replicate(3000, g(n)))
  qu <- quantile(pm, c(0.025, 0.975))
  list(q.025 = qu[1], q.975 = qu[2])
}
if(file.exists('real-lim.rds')) w <- readRDS('real-lim.rds') else {
  w <- data.table(n=10:500)
  w <- w[, tlim(n), by=.(n)]
  saveRDS(w, 'real-lim.rds')
}
w[, width := q.975 - q.025]

See if confidence limit widths following the law.

require(rms)
options(prType='html')
f <- ols(log(width) ~ log(n), data=w)
f

Linear Regression Model

ols(formula = log(width) ~ log(n), data = w)

	Model Likelihood Ratio Test	Discrimination Indexes
Obs 491	LR χ² 2808.90	R² 0.997
σ 0.0252	d.f. 1	R²_adj 0.997
d.f. 489	Pr(>χ²) 0.0000	g 0.468

Residuals

       Min         1Q     Median         3Q        Max 
-0.0679417 -0.0161243 -0.0004604  0.0140096  0.1499036

	β	S.E.	t	Pr(>\|t\|)
Intercept	2.6615	0.0073	362.20	<0.0001
n	-0.5290	0.0014	-385.65	<0.0001

p <- exp(predict(f))
w[, pwidth := p]
ggplot(w, aes(x=n, y=width)) + geom_smooth(se=FALSE) +
  geom_smooth(mapping=aes(x=n, y=pwidth), se=FALSE, color=I('red'))

Exact widths are in blue, and true CI widths predicted by a log-log model are in red. There is excellent agreement. The fitted model to estimate the 0.95 CI width is approximately , the last term being very close to . This formula can be used for planning studies when the goal is to achieve a specified precision (half-width of the CI) in estimating the true pseudomedian when the population distribution is our lognormal distribution. For other distributions just replace the g function above with one specifying the distribution of interest, and use the simulated widths as a function of to solve for the minimum yielding adequate precision (margin of error).

Note the similarity with the basic bootstrap.

Summary

The mean may be thought of as a “parametric quantity” and at present there is no general-purpose method for computing completely accurate (in both tails) confidence limits for means drawn from very asymmetric distributions. More methods for computing confidence limits for means are needed if you are likely to encounter very skewed distributions. The best available method at present is the BCa bootstrap, although its intervals are longer than some other methods.

The robustness and efficiency of the pseudomedian and the high accuracy of confidence limits for it (especially the bootstrap BCa and percentile limits) should make it more of a go-to measure of central tendency, for both symmetric and asymmetric distributions.

I hope that others will adapt the code provided here to study more sample sizes and distributions.

Confidence Intervals for the Mean Under Skewness: Why Every Fix We Tried Hit the Same Wall

The following was developed in an interaction with Claude Sonnet 4.6. The full chat with code and simulation results may be found here.

Starting point

We built ftboot(), an efficient R implementation of the fast triple bootstrap (FTB; Davidson & Trokić, 2020) for confidence intervals on the mean of a numeric vector. FTB approximates the (very expensive) standard triple bootstrap by replacing nested resampling with a single chain of three paired resamples per replicate, then inverting the resulting algorithm’s nested order-statistic lookups in closed form rather than by numerical root-finding. The implementation was validated against a literal, unvectorized version of the published algorithm and showed good coverage and fast runtimes under ordinary (e.g., normal) data.

The test that broke it

We then ran ftboot() on samples of size n=50 from a Lognormal(μ=0, σ=1) distribution, tracking left- and right-tail non-coverage separately (rather than just overall coverage, which can hide an imbalance between the tails). The result: the left tail was on target, but the right tail showed roughly double the nominal non-coverage rate. This matched what a plain BCa bootstrap also showed on the same data — and both findings echo Frank Harrell’s much larger study (Measures of Central Tendency for an Asymmetric Distribution, and Confidence Intervals, fharrell.com/post/aci), which found that across 18 procedures for the mean (including CLT-based, several bootstrap variants, and ad hoc skewness corrections), BCa was the best performer yet still had right-tail non-coverage of 0.086 against a nominal 0.025 at n=200 for an even more skewed lognormal, with no method tested closing the gap even at n=400.

What we tried, and why each attempt failed

Skewness-adjusted bootstrap-t. We tried layering a Johnson-style Edgeworth correction onto the studentized statistics feeding FTB’s resampling chain, in two variants: adjusting only the bootstrap reference distribution, and adjusting both the reference distribution and the observed pivot. Neither helped; the first made the right tail meaningfully worse. The reason: Johnson’s correction is designed to patch the gap between a t-statistic’s true distribution and an idealized symmetric reference table — but the bootstrap-t already replaces that idealized reference with the empirical bootstrap distribution, which already encodes the data’s skewness directly. Layering an analytic correction on top doesn’t add information; it distorts an already-correct empirical picture, in this case by uniformly shifting the whole reference distribution in a way that narrowed the right tail rather than widening it.

Switching the bootstrap variant (basic, percentile) instead of BCa. Reasoned through rather than tested: both are expected to perform similarly or worse than BCa under skew, since BCa’s whole design is to correct for exactly this asymmetry, and the simpler variants have no such correction at all.

Transform, bootstrap, back-transform. Bootstrapping log(x) and exponentiating the resulting interval endpoints gives a good CI for the median of X (or more precisely for exp of the log-scale mean), not for the mean of X — because the mean of a lognormal depends on both the log-scale mean and the log-scale variance, and naively exponentiating endpoints doesn’t propagate the variance term’s own uncertainty. This is a tool for a different estimand, not a shortcut to the same one.

A flexible 3–4 parameter distributional family (e.g., GB2, generalized gamma, Tukey’s g-h), fit by maximum likelihood, with the mean derived from the fitted parameters. This relocates the problem rather than solving it: the mean’s value under any of these families depends sensitively on the tail-shape parameter, and that parameter is estimated almost entirely from the few most extreme observations — the same scarce information that was starving the bootstrap’s right tail in the first place. The family’s flexibility helps only to the extent it is correctly specified, and verifying correct specification is hardest exactly where heavy tails make it matter most.

The common thread

Every method above is, in a structural sense, doing the same thing: extrapolating into a region of the distribution that a finite, skewed sample under-represents. Nonparametric methods (bootstrap, empirical likelihood) are bound by the support of the n observed values — they cannot produce a tail more extreme than what was actually drawn. Parametric methods relocate that same limitation into the precision of an estimated tail-shape parameter, which is no better determined by a small sample than the support itself is. Frank Harrell independently found empirical likelihood gave “terrible” coverage for the same kind of problem, consistent with this diagnosis: it is the same ceiling, not a defect specific to bootstrapping or to EL.

The deeper reason is that the mean is a linear functional of the data — every observation, however extreme, contributes to it with full weight and no downweighting. A single far-out draw moves the mean by an amount proportional to its own size; nothing in the definition of the mean offers a cushion against that. Estimating it well under skew therefore requires getting the tail right, and getting the tail right from a finite sample is hard in direct proportion to how heavy and asymmetric that tail is. This is true whether the tail-fitting is done implicitly (resampling) or explicitly (a fitted shape parameter); the requirement doesn’t go away, only the vocabulary used to describe it changes.

The resolution: change the estimand, not the method

The median and the pseudomedian (Hodges-Lehmann estimator) are defined by rank/order information rather than by a weighted sum. The median doesn’t care how extreme the largest observation is, only that it’s larger than the others — so its sampling distribution stays close to symmetric and well-behaved even when the population is heavily skewed. This is a difference in kind, not degree: these estimators were never exposed to the tail-extrapolation problem that the mean cannot escape.

This matches Harrell’s finding in the linked post: confidence intervals for the pseudomedian — both the BCa bootstrap version and the exact Hodges-Lehmann interval (which only assumes symmetry of the underlying distribution, not any particular parametric shape) — achieved near-nominal non-coverage in both tails at n=200 under the same lognormal distribution where 18 different procedures for the mean all fell short. The pseudomedian also retains most of the mean’s efficiency under normality (95.5% relative efficiency) while having a breakdown point of 0.29, versus the mean’s breakdown point of 0.

Bottom line

For a continuous distribution of genuinely unknown and possibly asymmetric, heavy-tailed shape, no confidence interval procedure for the mean — bootstrap-based, analytically corrected, or built on a flexible parametric family — can be expected to reliably get both tails right at realistic sample sizes, because the difficulty is inherent to what the mean is, not to any particular method of estimating it. The pseudomedian (or median) sidesteps this problem entirely rather than mitigating it, which is the core argument for treating it as a primary measure of central tendency even when the data-generating distribution is not known to be symmetric.

References

Davidson, R. and Trokić, M. (2020). “The Fast Iterated Bootstrap.” Journal of Econometrics, 218(2), 451–475.
Harrell, F. (2025). “Measures of Central Tendency for an Asymmetric Distribution, and Confidence Intervals.” Statistical Thinking. https://fharrell.com/post/aci

Computing Environment

grateful::cite_packages(pkgs='Session', output='paragraph', out.dir='.',
    cite.tidyverse=FALSE, omit=c('grateful', 'ggplot2'))

We used R v. 4.6.0 (R Core Team 2026) and the following R packages: bayesboot v. 0.2.3 (Bååth 2025), boot v. 1.3.32 (Davison and Hinkley 1997; Canty and Ripley 2025), data.table v. 1.18.4 (Barrett et al. 2026), Hmisc v. 5.2.6 (Harrell Jr 2026a), rms v. 8.1.2 (Harrell Jr 2026b).

The code was run on macOS Tahoe 26.5.1 on a Macbook Pro M2 Max.

References

Bååth, Rasmus. 2025. bayesboot: An Implementation of Rubin’s (1981) Bayesian Bootstrap. https://doi.org/10.32614/CRAN.package.bayesboot.

Barrett, Tyson, Matt Dowle, Arun Srinivasan, et al. 2026. data.table: Extension of “data.frame”. https://doi.org/10.32614/CRAN.package.data.table.

Canty, Angelo, and Brian Ripley. 2025. boot: Bootstrap Functions. https://doi.org/10.32614/CRAN.package.boot.

Davison, Anthony Christopher, and David Victor Hinkley. 1997. Bootstrap Methods and Their Applications. Cambridge University Press. https://doi.org/10.1017/CBO9780511802843.

Harrell Jr, Frank E. 2026a. Hmisc: Harrell Miscellaneous. https://hbiostat.org/R/Hmisc/.

Harrell Jr, Frank E. 2026b. rms: Regression Modeling Strategies. https://hbiostat.org/R/rms/.

R Core Team. 2026. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing. https://doi.org/10.32614/R.manuals.

Reuse

CC BY 4.0

Bootstrap Confidence Limits for Bootstrap Overfitting-Corrected Model Performance

Frank Harrell — Thu, 24 Jul 2025 05:00:00 GMT

Background

The goal here is strong internal validation after fitting a pre-specified regression model or one that was derived using backwards step-down variable selection such that the same variable selection procedure can be repeated afresh for each bootstrap repetition. So strong internal validation means estimating a variety of model performance measures in a way that does not reward them for overfitting and that penalizes for all aspects of model selection and derivation that utilized the outcome variable. The goal is estimating the likely future performance of the model on new data from the same stream as the data used in model development. A standard approach, as implemented in the R rms package’s validate and calibrate functions, is the Efron-Gong optimism bootstrap discussed in detail in RMS.

A key omission from the rms package has been computation of uncertainty intervals for bootstrap overfitting-corrected measures. In 2021 Hisashi Noma et al. wrote a very useful paper entitled Confidence intervals of prediction accuracy measures for multivariable prediction models based on the bootstrap-based optimism correction methods. They studied confidence interval coverage (although only in a two-tailed combined sense) of a full two-stage double bootstrap procedure to compute confidence intervals for Efron-Gong overfitting (bias/optimism)-corrected model performance measures. This is a very computationally demanding but accurate procedure. They also studied an approximate one-stage procedure that does not add to the computational burden of computing the overfitting-adjusted measures themselves.

Recall that the form of the bootstrap optimism-corrected measure for a single index such as a Brier score or rank correlation between predicted and observed outcomes is

where is the original index computed as usual on the whole sample (sometimes called the apparent index), is the apparent performance in a bootstrap sample when model coefficients are fitted in the bootstrap sample, is the performance of the bootstrap-fitted model on the original whole sample, and the horizontal bars represent averages over bootstrap resamples. The estimated bias is .

Noma et al.’s approximate bootstrap confidence intervals take the bootstrap distribution of , computes confidence intervals from it using the bootstrap percentile nonparametric method, and shifts the resulting intervals by . They showed that in large samples, taking into account only the variation in is good enough, and that the overall confidence interval coverage was close to that of the expensive double bootstrap. But when the sample size is not large, considering only variation in significantly underestimates uncertainties, and confidence interval coverage is far below the intended level (of 0.95, for example).

Fast Alternative Bootstrap Confidence Limits

To derive a fast approximate method that I expect to have better confidence coverage for non-large , consider individual bootstrap sample estimates and and compute variances of several combinations of them, including , , , variance of weighted sums of and , and other variants. Variances of vectors involving both and takes into account uncertainties in two quantities instead of one. One can think of the variance of a difference as a summative measure for two uncertainties, because the variance of a difference is the sum of the variances minus twice the covariance of the two. It is tempting to use the bootstrap nonparametric percentile interval, but we need to respect the width of the bootstrap distribution for the combination of and and pay no attention to its center. This pushes us to use normal-approximation bootstrap confidence intervals, which take an original point estimate (here a bootstrap bias-corrected index) the bootstrap standard deviation, for example.

An additional complexity is that the bootstrap distribution may not be symmetric, and assuming that it is (as the standard deviation does) leads to symmetric confidence intervals instead of the needed asymmetric ones. The result is inaccurate confidence coverage in at least one of the tails. To fix that problem, we compute two standard deviations: the SD for upper values and the SD for lower ones. There is a question of where to make the high/low split. Traditionally the sample mean is chosen. Each of the two confidence limits is computed using the square root of the average squared distance from the overall mean of the bootstrap distribution.

Next we attempt to find the best combination of variation of training and test indexes from bootstrap repetitions so that confidence interval coverage for overfitted-corrected indexes is good in both tails for three performance measures. The resulting bootstrap method will be called ABCLOC for asymmetric bootstrap confidence limits for overfitting-corrected model performance metrics.

Simulation to Derive ABCLOC to Obtain Accurate Confidence Coverage for Simple Performance Measures

Before proceeding to a more complex situation of studying accuracy of confidence intervals for estimated overfitting-corrected calibration curves, let’s study the performance of ABCLOC confidence limits for standard scalar predictive accuracy measures: Somers’ (equal to where is the concordance probability or area under the ROC curve), the calibration slope on the logit scale, and the Brier score. The R rms package validate.lrm function uses, among other methods, the Efron-Gong optimism bootstrap for debiasing the apparent performance indexes to obtain overfitting-corrected scores estimating likely future performance of the model for these three, and other indexes.

Define a function to simulate a matrix of p predictors for a sample of size , and a sample of size of binary responses from that predictor matrix, using a true model where the only non-zero regression coefficient for the predictors is .

require(Hmisc)
require(rms)   # Version 8.1-0 is required
require(ggplot2)
require(data.table)
options(prType='html')

simd <- function(n, p) {
  x <- matrix(rnorm(n * p), n, p)
  L <- x[, 1]
  y <- ifelse(runif(n) <= plogis(L), 1, 0)
  list(x=x, y=y)
}

In what follows let be the sample size and be the number of predictors. This is a setting with major overfitting.

To check confidence interval coverage we need to know the true Brier score for our data generating model. This can be well estimated by simulating a single sample of 200,000 subjects, and computing the Brier score for many models fitted on 200 subjects, averaging their Brier scores.

set.seed(1)
N    <- 200
np   <- 15
Nbig <- 200000
big  <- simd(Nbig, np)
big  <- data.frame(x = I(big$x), y=big$y)   # I keeps matrix x as a matrix
Y    <- big$y
nsim <- 5000

Simulate 5000 samples of size 200. For each sample run 300 bootstrap resamples to compute approximate confidence limits. Also compute the true performance measures for each sample, to average.

# Define a function to generate and analyze one dataset
sim1 <- function(n, p, B=300) {
  .i. <<- .i. + 1
  cat(.i., '', file='/tmp/z', append=TRUE)
  d    <- simd(n, np)
  f    <- lrm(y ~ x, data=d, x=TRUE, y=TRUE)
  phat <- predict(f, newdata=big, type='fitted')
  
  trueB     <- mean((phat - Y) ^ 2)
  trueDxy   <- somers2(phat, Y)['Dxy']
  trueSlope <- lrm.fit(qlogis(phat), Y)$coef[2]
  # saveraw is passed to predab.re-sample in rms 8.1-0 to cause indexes
  # from individual bootstrap resamples to be saved in a global object .predab_raw.
  v <- validate(f, B=B, saveraw=TRUE)
  r <- .predab_raw.
  w <- which(names(r$orig) %in% c('Dxy', 'Slope', 'B'))
  r$orig   <- r$orig[w]
  r$btrain <- r$btrain[, w]
  r$btest  <- r$btest [, w]
  praw[[.i.]] <<- r
  v <- v[w, c('index.corrected', 'Lower', 'Upper')]
  v <- cbind(v, true=c(trueDxy, trueSlope, trueB))
  v
}
.i.  <- 0
praw <- NULL
set.seed(1)
if(file.exists('indexes.rds')) {
  s    <- readRDS('indexes.rds')
  praw <- readRDS('praw.rds')
  } else {
  s <- replicate(nsim, sim1(n=N, p=np))
  saveRDS(s,    'indexes.rds')
  saveRDS(praw, 'praw.rds')
}

# Compute means across repetitions  
rn <- function(x) round(x, 3)
rn(apply(s, 1:2, mean))

      index.corrected Lower Upper  true
Dxy             0.426 0.303 0.550 0.404
Slope           0.680 0.407 0.997 0.636
B               0.224 0.191 0.263 0.226

The average overfitting-corrected Brier score estimate closely tracks the average of all the true Brier scores. The optimism bootstrap overestimated the performance on the and calibration slope scales. The positive bias is 0.02 for and 0.04 for the calibration slope.

It is known that in extremely overfitted models, the bootstrap will underestimate the amount of overfitting when compared with 100 repeats of 10-fold cross-validation. But even in these situations, model performance estimated by the bootstrap will convey enough bad news.

Next examine tail non-coverage probabilities for a variety of ways of utilizing the training and test indexes over the 300 resamples drawn for each simulated dataset. This is done separately for each of the 3 performance indexes.

# Add true indexes to praw
raw <- praw
for(i in 1 : nsim) raw[[i]]$true <- s[, 'true', i]

# Function applied to each element w of raw, for bootstrap type and basis and
# index number idx
# Result is a pair of logical values left and right with TRUE indicating that
# the true index is outside that tail

h <- function(w, what=c('error', 'cltrue')) {
  what  <- match.arg(what)
  true  <- w$true [idx]
  train <- w$btrain[, idx]
  test  <- w$btest [, idx]
  opt   <- train - test
  mopt  <- mean(opt)
  theta <- w$orig[idx]
  x <- switch(basis,
              train = train,
              test  = test,
              opt   = opt,
              wtd   = 2    * train - test,
              wtd2  = 1.5  * train - test,
              wtd3  = train - 0.75 * test,
              wtd4  = train - 1.25 * test,
              wtd5  = train - 1.5 * test)
  
  if(type == 'shifted nonpar') {
    qu <- quantile(x, c(0.025, 0.975), na.rm=TRUE)
    r <- c(qu[1] - mopt, qu[2] - mopt)
    } else if(substr(type, 1, 2) == 'sd') {
    if(basis == 'v2') {
      v1 <- var(train)
      v2 <- var(train - test)
      s  <- sqrt(v1 + v2 - 0.5 * sqrt(v1 * v2))
      r <- c(theta - mopt - s * z, theta - mopt + s * z)
    }  else {   # dualSD is in Hmisc 5.2-4
    s <- dualSD(x, nmin=if(type == 'sd1') 100000 else 10)
    a <- if(type == 'sd2') 1:2 else 2:1
    r <- c(theta - mopt - s[a[1]] * z, theta - mopt + s[a[2]] * z)
    }
    }
  if(what == 'cltrue') return(c(left=r[1], right=r[2], true=true))
  r <- c(r[1] > true, r[2] < true)
  names(r) <- .q(left, right)
  r
  }

z  <- qnorm(0.975)

# validate used wtd4 sd2rev; check against that for Brier score
if(FALSE) {
  idx  <- 3
  type <- 'sd2rev'
  basis <- 'wtd4'
  for(i in c(1, nsim / 2, nsim)) {
    cat('i=', i, '\n')
    print(h(raw[[i]], 'cltrue'))
    print(s[3,,i])
  }
}

types <- c('shifted nonpar', 'sd1', 'sd2', 'sd2rev')
bases <- c('train', 'test', 'opt', 'wtd', 'wtd2', 'wtd3', 'wtd4', 'wtd5', 'v2')

D <- vector('list', 3)
for(idx in 1 :3) {
  idxname <- c('Dxy', 'Slope', 'B')[idx]
  cat('\n-------------------------------------------------------------------\nIndex: ',
      if(idxname == 'B') 'Brier' else idxname, '\n\n', sep='')

  # Analyze simulation results separately for each confidence interval construction method
  # For each simulation compute the components of confidence intervals
  # Then compute the per-simulation limits
  # Then compute for that simulation whether a non-coverage has occurred per tail

  d <- NULL
  for(type in types) {
    for(basis in bases) {
      if(type == 'shifted nonpar' && basis %nin% c('train', 'test')) next
      if(type != 'sd1'            && basis == 'v2') next
      w    <- sapply(raw, h)
      err  <- apply(w, 1, mean, na.rm=TRUE)
      dist <- sum(abs(err - 0.025))
      d <- rbind(d, data.frame(type, basis, left=rn(err[1]), right=rn(err[2]),
                               coverage=rn(1 - sum(err)), dist=rn(dist)))
    }
  }
  row.names(d) <- NULL
  print(d)
  D[[idx]] <- d
}


-------------------------------------------------------------------
Index: Dxy

             type basis  left right coverage  dist
1  shifted nonpar train 0.350 0.001    0.649 0.349
2  shifted nonpar  test 0.122 0.474    0.404 0.546
3             sd1 train 0.124 0.033    0.843 0.107
4             sd1  test 0.355 0.156    0.489 0.461
5             sd1   opt 0.122 0.035    0.842 0.108
6             sd1   wtd 0.010 0.001    0.989 0.039
7             sd1  wtd2 0.043 0.006    0.951 0.037
8             sd1  wtd3 0.128 0.037    0.836 0.114
9             sd1  wtd4 0.114 0.032    0.853 0.097
10            sd1  wtd5 0.103 0.028    0.870 0.080
11            sd1    v2 0.073 0.017    0.910 0.057
12            sd2 train 0.112 0.034    0.854 0.096
13            sd2  test 0.314 0.184    0.502 0.448
14            sd2   opt 0.115 0.036    0.848 0.102
15            sd2   wtd 0.007 0.001    0.992 0.042
16            sd2  wtd2 0.038 0.006    0.955 0.032
17            sd2  wtd3 0.120 0.038    0.842 0.108
18            sd2  wtd4 0.111 0.032    0.857 0.093
19            sd2  wtd5 0.104 0.026    0.870 0.080
20         sd2rev train 0.133 0.032    0.835 0.115
21         sd2rev  test 0.393 0.129    0.478 0.472
22         sd2rev   opt 0.128 0.034    0.838 0.112
23         sd2rev   wtd 0.013 0.001    0.986 0.036
24         sd2rev  wtd2 0.048 0.005    0.947 0.043
25         sd2rev  wtd3 0.135 0.035    0.830 0.120
26         sd2rev  wtd4 0.116 0.032    0.853 0.097
27         sd2rev  wtd5 0.103 0.029    0.868 0.082

-------------------------------------------------------------------
Index: Slope

             type basis  left right coverage  dist
1  shifted nonpar train 0.661 0.339    0.000 0.950
2  shifted nonpar  test 0.000 0.508    0.492 0.508
3             sd1 train 0.661 0.339    0.000 0.950
4             sd1  test 0.065 0.045    0.890 0.060
5             sd1   opt 0.065 0.045    0.890 0.060
6             sd1   wtd 0.065 0.045    0.890 0.060
7             sd1  wtd2 0.065 0.045    0.890 0.060
8             sd1  wtd3 0.170 0.076    0.754 0.196
9             sd1  wtd4 0.015 0.026    0.958 0.011
10            sd1  wtd5 0.002 0.015    0.983 0.033
11            sd1    v2 0.065 0.045    0.890 0.060
12            sd2 train 0.661 0.339    0.000 0.950
13            sd2  test 0.088 0.037    0.875 0.075
14            sd2   opt 0.048 0.054    0.897 0.053
15            sd2   wtd 0.048 0.054    0.897 0.053
16            sd2  wtd2 0.048 0.054    0.897 0.053
17            sd2  wtd3 0.146 0.089    0.765 0.185
18            sd2  wtd4 0.010 0.035    0.955 0.025
19            sd2  wtd5 0.002 0.021    0.977 0.027
20         sd2rev train 0.661 0.339    0.000 0.950
21         sd2rev  test 0.048 0.054    0.897 0.053
22         sd2rev   opt 0.088 0.037    0.875 0.075
23         sd2rev   wtd 0.088 0.037    0.875 0.075
24         sd2rev  wtd2 0.088 0.037    0.875 0.075
25         sd2rev  wtd3 0.199 0.067    0.735 0.215
26         sd2rev  wtd4 0.024 0.020    0.956 0.006
27         sd2rev  wtd5 0.004 0.011    0.985 0.035

-------------------------------------------------------------------
Index: Brier

             type basis  left right coverage  dist
1  shifted nonpar train 0.001 0.303    0.696 0.302
2  shifted nonpar  test 0.626 0.060    0.315 0.635
3             sd1 train 0.047 0.067    0.886 0.064
4             sd1  test 0.212 0.246    0.542 0.408
5             sd1   opt 0.016 0.045    0.938 0.029
6             sd1   wtd 0.000 0.001    0.999 0.049
7             sd1  wtd2 0.001 0.009    0.989 0.039
8             sd1  wtd3 0.025 0.051    0.924 0.026
9             sd1  wtd4 0.010 0.038    0.952 0.028
10            sd1  wtd5 0.006 0.031    0.963 0.025
11            sd1    v2 0.009 0.026    0.965 0.017
12            sd2 train 0.038 0.073    0.889 0.061
13            sd2  test 0.248 0.211    0.541 0.409
14            sd2   opt 0.009 0.055    0.936 0.047
15            sd2   wtd 0.000 0.003    0.997 0.047
16            sd2  wtd2 0.001 0.013    0.987 0.037
17            sd2  wtd3 0.015 0.063    0.921 0.048
18            sd2  wtd4 0.005 0.049    0.947 0.044
19            sd2  wtd5 0.002 0.041    0.957 0.039
20         sd2rev train 0.060 0.060    0.880 0.070
21         sd2rev  test 0.174 0.280    0.545 0.405
22         sd2rev   opt 0.028 0.036    0.936 0.014
23         sd2rev   wtd 0.000 0.001    0.999 0.049
24         sd2rev  wtd2 0.004 0.007    0.989 0.039
25         sd2rev  wtd3 0.034 0.043    0.922 0.028
26         sd2rev  wtd4 0.021 0.029    0.950 0.008
27         sd2rev  wtd5 0.013 0.022    0.965 0.015

For each bootstrap confidence interval method and each performance index, compute the sum of two distances from the target tail probabilities of 0.025, and then sum these distances over the three indexes and see which method has the best overall confidence interval coverage accuracy.

# Sum distances over the 3 indexes
di <- D[[1]]$dist + D[[2]]$dist + D[[3]]$dist
cbind(distance=di, rank=rank(di))

      distance rank
 [1,]    1.601 26.0
 [2,]    1.689 27.0
 [3,]    1.121 24.0
 [4,]    0.929 20.0
 [5,]    0.197 14.0
 [6,]    0.148 10.0
 [7,]    0.136  5.5
 [8,]    0.336 17.0
 [9,]    0.136  5.5
[10,]    0.138  7.0
[11,]    0.134  4.0
[12,]    1.107 23.0
[13,]    0.932 22.0
[14,]    0.202 16.0
[15,]    0.142  8.0
[16,]    0.122  2.0
[17,]    0.341 18.0
[18,]    0.162 13.0
[19,]    0.146  9.0
[20,]    1.135 25.0
[21,]    0.930 21.0
[22,]    0.201 15.0
[23,]    0.160 12.0
[24,]    0.157 11.0
[25,]    0.363 19.0
[26,]    0.111  1.0
[27,]    0.132  3.0

The winner (rank of 1) is sd2rev wtd4 which computes standard deviations of the 300 bootstrap re-sampled values of training - 1.25 test indexes, where training denotes apparent performance on the bootstrap sample and test denotes the bootstrap fit’s performance on the original whole sample. The simple SD of the 300 estimates of bias (optimism; sd1* opt) had overall ranks of 14, 16, and 15. It stands to reason that the best quantity on which to compute SDs puts more weight on the test sample indexes than it does on the training sample, because there is less variation in performance of bootstrap fits on the original sample over simulated datasets.

sd2rev refers to using the “top SD” for the lower confidence limit and the “bottom SD” for the upper confidence limit. The distance rank for using the ordinary SD was 5.5 and the rank when reversing the two directional SDs was 13. The standard parametric bootstrap using approximate normality does not handle asymmetric sampling distributions, but bottom and top SDs do. Better performance using two SDs indicates asymmetry of sampling distributions for the indexes.

Coverage for calibration slope and Brier score was estimated to be 0.955 and 0.947 for the best overall performing method. It did not perform so well for , giving only 0.852 coverage. has a somewhat degenerate distribution when computed on regression fits, which typically do not allow , so normality is usually not a good fit for .

Accuracy of Estimated Calibration Curves and Confidence Limits

Let’s study how well the overall winning bootstrap confidence interval estimator for three regular performance indexes fares when computing confidence bands for a vector of parameters, bootstrap overfitting-corrected calibration curves. Let’s fix the grid of 50 equally spaced points for which the calibration curve is estimated, using the 0.02 and 0.98 quantiles of pooled distributions of predicted probabilities for 10 repeated model fits. The simulations we are about to run are similar to those for the 3 statistical indexes, but now we have 50 indexes.

A histogram of the pooled distribution shows that the entire range of predicted probabilities is represented by the chosen logistic model and distribution of .

set.seed(1)
phat <- replicate(10, {
  d <- simd(N, np)
  f <- lrm(y ~ x, data=d)
  predict(f, type='fitted')
})
round(apply(phat, 2, quantile, c(0, 0.02, 0.98, 1)), 2)

     [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]
0%   0.03 0.03 0.01 0.07 0.03 0.02 0.08 0.02 0.02  0.05
2%   0.07 0.09 0.04 0.09 0.07 0.06 0.12 0.07 0.06  0.07
98%  0.95 0.94 0.96 0.90 0.93 0.94 0.84 0.86 0.90  0.93
100% 0.99 0.96 0.99 0.97 0.98 0.98 0.86 0.95 0.97  0.97

hist(phat, nclass=50, main=NULL, xlab=expression(hat(P)))

# On average for n=200, request calibrated probabilities for 5th to 196th
# largest observed predicted risks, which is pushing things a bit
r <- quantile(phat, probs=c(0.02, 0.98))
cat('Limits on predicted probabilities for which calibration curve is estimated:',
    round(r, 3), '\n')

Limits on predicted probabilities for which calibration curve is estimated: 0.075 0.933

pp <- seq(r[1], r[2], length=50)

For a given logistic model fit f on dataset d compute the true calibration curve at pp. The calibration curve is a set of points , where are predicted probabilities and are true probabilities that given . The points are fitted using least squares in the logit (the true calibration is linear in the logits by design) to average over variation in that yields unnecessary variation in that arises from giving the last (noise) ’s a chance to get nonzero ’s. So the true calibration curve is an average over covariate distributions, conditioning only on the covariates through conditioning on .

truecal <- function(f, d, pl=FALSE) {
  Lhat  <- predict(f)
  Ltrue <- d$x[, 1]
  phat  <- plogis(Lhat)
  int_slope <- lm.fit.qr.bare(Lhat, Ltrue)$coefficients
  ptrue <- plogis(Ltrue)
  if(pl) plot(phat, ptrue, col='gray80')
  plogis(int_slope[1] + int_slope[2] * qlogis(pp))
}

# Test the function
par(mfrow=c(3,3), mar=c(2,2,0,0))
set.seed(3)
g <- function() {
  d <- simd(N, np)
  f <- lrm(y ~ x, data=d)
  tcal <- truecal(f, d, pl=TRUE)
  lines(pp, tcal)
  abline(a=0, b=1, col='gray80')
  coef(f)
}
round(replicate(9, g()), 2)

           [,1]  [,2]  [,3]  [,4]  [,5]  [,6]  [,7]  [,8]  [,9]
Intercept  0.28  0.00 -0.02  0.07  0.13 -0.07  0.02  0.24 -0.24
x[1]       0.86  1.13  0.98  1.10  1.28  1.10  0.93  0.77  1.27
x[2]      -0.10  0.02  0.00  0.29  0.07  0.21  0.01 -0.03  0.15
x[3]       0.18  0.28  0.04  0.42  0.24  0.24  0.09 -0.19 -0.43
x[4]      -0.06  0.22  0.21 -0.10 -0.17 -0.05  0.21  0.54  0.16
x[5]      -0.12  0.01  0.08  0.02 -0.12 -0.23  0.20  0.12  0.35
x[6]      -0.07 -0.09  0.22  0.06 -0.53  0.16  0.44  0.05 -0.21
x[7]      -0.43 -0.12  0.09  0.14 -0.15 -0.32  0.15  0.06 -0.31
x[8]       0.12 -0.18 -0.09  0.00  0.12  0.38 -0.15 -0.19  0.07
x[9]      -0.07  0.12 -0.25 -0.21 -0.12 -0.07 -0.03 -0.09 -0.27
x[10]     -0.22 -0.04  0.08  0.03  0.03  0.12  0.00  0.30  0.19
x[11]     -0.12  0.03  0.09 -0.05 -0.03  0.01  0.03  0.16  0.38
x[12]      0.06  0.09 -0.26 -0.11 -0.34  0.03 -0.01 -0.17 -0.03
x[13]      0.15 -0.19 -0.15  0.34  0.21  0.06 -0.18  0.09 -0.28
x[14]      0.06  0.26  0.07  0.29 -0.01 -0.22 -0.23 -0.03  0.04
x[15]      0.00 -0.07  0.01  0.01  0.42  0.17 -0.01 -0.22 -0.01

The fact that the circles are sometimes far from the curves implies that an “average” true calibration curve is hiding some false variation in predicted risks due to variation in the extraneous predictors. In other words, many different predicted values (depending on the noise predictors) can give rise to one true probability. To have a gold standard to compare with bootstrap estimates we must take averages.

The default calibration estimator used in calibrate() is loess, using the R lowess function with outlier detection turned off (iter=0).

Define alternative parametric linear and quadratic (in the logit) estimators.

qcal <- function(x, y, deg=2) {
  # Curtail predicted probabilities to [0.001, 0.999]
  x <- pmax(0.001, pmin(x, 0.999))
  L <- qlogis(x)
  f <- lrm.fit(if(deg == 2) cbind(L, L^2) else L, y, compstats=FALSE)
  k <- coef(f)
  pgrid <- seq(min(x), max(x), length=200)
  Lgrid <- qlogis(pgrid)
  cgrid <- k[1] + k[2] * Lgrid
  if(deg == 2) cgrid <- cgrid + k[3] * Lgrid^2
  cal   <- plogis(cgrid)
  list(x=pgrid, y=cal)
}

Test the linear and quadratic calibrations in a setting where the calibration curve is known to be the line of identity, and also include loess estimates (red). Linear logistic calibration uses a black line, and quadratic a blue line.

par(mfrow=c(3, 3), mar=c(2, 2, 0, 0))
set.seed(1)
for(i in 1 : 9) {
  x <- runif(N)
  y <- ifelse(runif(N) <= x, 1, 0)
  w1 <- qcal(x, y, deg=1)
  plot(w1, type='l', xlim=c(0, 1), ylim=c(0, 1))
  w2 <- qcal(x, y, deg=2)
  lines(w2, col='blue')
  l <- lowess(x, y, iter=0)
  lines(l, col='red')
  abline(a=0, b=1, col='grey80')
}

Define a function that draws one sample, fits a binary logistic model, and computes bootstrap percentile confidence bands for the calibration curve at a regular sequence of predicted probabilities pp. Add the true calibration curve to the output, alongside , where is number of observed predicted risks that are less than the current value of pp and is the number of that are greater than the value of pp. In addition, save the standard deviation of . This is done for one calibration estimator. A special case is smoother='basic' which stands for using the validate function to get bias-corrected intercept and slope estimates and to convert them to linear (in the logit) calibration curves, without confidence limits. Finally, smoother='apparent' does not use bootstrapping and instead takes the to (falsely) need no calibration, to gauge the accuracy of uncalibrated estimates.

# Use the following if you want to count "close" observations
# pptab <- data.table(pp, low = pp - 0.05, hi = pp + 0.05)
pptab <- data.table(ppgrid = pp)
sim1 <- function(n, p, B=300, smoother='lowess', pl=FALSE) {
  .i. <<- .i. + 1
  cat(.i., '', file='/tmp/z', append=TRUE)
  d <- simd(n, p)
  f <- lrm(y ~ x, data=d, x=TRUE, y=TRUE)
  phat    <- predict(f, type='fitted')
  ptrue   <- truecal(f, d)
  if(is.character(smoother) && smoother != 'lowess') {
    switch(smoother,
           basic = {
            val       <- validate(f, B=B)[, 'index.corrected']
            int_slope <- val[c('Intercept', 'Slope')]
            estcal    <- plogis(int_slope[1] + int_slope[2] * qlogis(pp)) },
           apparent = {
             estcal   <- pp } )
    lower <- NA
    upper <- NA
    if(pl) {
      plot(pp, estcal, type='l', xlim=c(0,1), ylim=c(0,1))
      lines(pp, ptrue, col='blue')
    }
  } else {
    bootcal <- calibrate(f, B=B, predy=pp, smoother=smoother)
    estcal <- bootcal[, 'calibrated.corrected']
    lower  <- pp + bootcal[, 'Lower']
    upper  <- pp + bootcal[, 'Upper']
    if(pl) {
      plot(bootcal, ylim=c(0,1))
      lines(pp, ptrue,   col='blue')
    }
  }
  d      <- data.table(phat)
  lo <- d[pptab, on = .(phat <= ppgrid), .N, by=.EACHI][, N]
  hi <- d[pptab, on = .(phat >= ppgrid), .N, by=.EACHI][, N]
  # count  <- d[pptab, on = .(phat >= low, phat <= hi), .N, by = .EACHI][, N]
  invisible(cbind(pp, estcal, ptrue, lower, upper, count=pmin(lo, hi), sphat=sd(phat)))
}

Run 9 tests for the default loess calibration estimator, plotting the confidence bands. The blue curve is the true calibration curve. In the upper left of each panel show the number of observations with low and high predicted probabilities.

set.seed(9)
par(mfrow=c(3,3), mar=c(2, 2, 0, 0))
for(j in 1 : 9) sim1(N, np, pl=TRUE)


n=200   Mean absolute error=0.047   Mean squared error=0.00321
0.9 Quantile of absolute error=0.101


n=200   Mean absolute error=0.039   Mean squared error=0.00196
0.9 Quantile of absolute error=0.07


n=200   Mean absolute error=0.041   Mean squared error=0.00284
0.9 Quantile of absolute error=0.089


n=200   Mean absolute error=0.042   Mean squared error=0.00341
0.9 Quantile of absolute error=0.11


n=200   Mean absolute error=0.06   Mean squared error=0.00434
0.9 Quantile of absolute error=0.09


n=200   Mean absolute error=0.042   Mean squared error=0.00274
0.9 Quantile of absolute error=0.084


n=200   Mean absolute error=0.056   Mean squared error=0.00361
0.9 Quantile of absolute error=0.088


n=200   Mean absolute error=0.046   Mean squared error=0.00271
0.9 Quantile of absolute error=0.077


n=200   Mean absolute error=0.039   Mean squared error=0.00315
0.9 Quantile of absolute error=0.089

Run 1000 simulations using all three calibration curve estimators. For each dataset simulated, record the number of observations with low/high predictions. Also compute the more direct estimated bias-corrected linear calibration curve by using the validate function to debias the intercept and slope computed from bootstrapping that is separate from the calibrate run.

sm <- list('lowess',
           function(x, y) qcal(x, y, deg=1),
           function(x, y) qcal(x, y, deg=2),
           'basic',
           'apparent')
set.seed(7)
if(file.exists('cal.rds')) R <- readRDS('cal.rds') else {
  R        <- vector('list', 5)
  names(R) <- c('loess', 'linear', 'quadratic', 'basic', 'apparent')
  for(est in 1 : 5) {
    cat('\nEstimator', est, '\n')
    .i. <- 0
    r <- replicate(1000, sim1(N, np, B=300, smoother=sm[[est]]))  # 44m
    R[[est]] <- r
  }
  saveRDS(R, 'cal.rds')
}

Restructure the results into a list of data tables.

convert2dt <- function(r) {
  npp  <- length(pp)
  nsim <- dim(r)[3]
  vn   <- dimnames(r)[[2]]
  # Restructure array so that pp and simulation # move fastest
  r <- aperm(r, c(1, 3, 2))
  # Stack into a taller matrix
  r <- matrix(r, nrow = npp * nsim, ncol=length(vn))
  d <- data.table(pp  = rep(pp, times=nsim),
                  sim = rep(1:nsim, each=npp) )
  d[, (vn) := as.data.table(r)]
  d
}
d <- lapply(R, convert2dt)

Accuracy of Bootstrap Calibration Estimates for

Group the number of points within 0.05 of the target estimated risk into intervals and compute pseudo-medians of absolute calibration curve estimation error by calibration estimator, target , and group, where groups are intervals of SDs or minimum tail frequencies.

The pseudo-median is the median of all possible pairwise means. It is more robust than the mean and is virtually as efficient as the mean, unlike the sample median.

pacc <- function(d, stratby) {
  # Restructure to a single tall and thin data table
  w <- rbindlist(d, idcol='estimator')
  if(! missing(stratby)) {
    w[, let(z         = abs(estcal - ptrue),
            group     = eval(stratby) ) ]
    w[, prn(table(group))]
    wall <- copy(w)
    wall[, group := 'All']
    w <- rbind(w, wall)
    w <- w[, .(err = pMedian(z, na.rm=TRUE)), by=.(estimator, group, pp)]
    g <- ggplot(w, aes(x=pp, y=err, color=estimator)) +
      facet_wrap(~ group)
  } else{
    w[, let(z = abs(estcal - ptrue))]
    w <- w[, .(err = pMedian(z, na.rm=TRUE)), by=.(estimator, pp)]
    g <- ggplot(w, aes(x=pp, y=err, color=estimator))
  }
  g + geom_line() + 
    xlab(expression(hat(P))) + ylab('pMedian |Calibration Estimation Error|')
}
pacc(d, expression(cut2(sphat, c(.175, .2, .25)))) + labs(caption='By groups of SD(Phat), n=200')


table(group)

group
[0.142,0.175) [0.175,0.200) [0.200,0.250) [0.250,0.361] 
         1700         11750        137000         99550

pacc(d, expression(cut2(count, 50))) + labs(caption='By groups of min(tail count), n=200')


table(group)

group
[  0, 50) [ 50,100] 
   134704    115296

It is typically true that even in heavily overfitted models, predictions that are near the grand mean estimated risk are accurate. This is reflected in the above plots which show the “no calibration” method of just trusting the apparent predicted risks to have the lowest expected error in the middle of the risk distribution. But it is highly overoptimistic for outer risks.

Run the following if you want to see a tensor spline surface for , SD of .

require(mgcv)
for(nam in names(d)) {
  w <- d[[nam]]
  f <- gam(abs(estcal - ptrue) ~ te(pp, sphat), data=w)
  plot(f, scheme=2, main=nam)
}

Accuracy of Approximate Bootstrap Confidence Limits for Calibration,

For each value of pp compute the proportion of simulations for which lower > ptrue and the proportion upper < ptrue (tail non-coverage proportions). Also compute overall error proportions.

plotit <- function(R, title) {
  left  <- R[, 'lower', ] > R[, 'ptrue', ]
  right <- R[, 'upper', ] < R[, 'ptrue', ]
  
  rmn <- function(x) round(mean(x, na.rm=TRUE), 3)
  cat('Overall non-coverage for', title, ':\tleft=', rmn(left),
      ' right=', rmn(right), '\n')

  left  <- apply(left , 1, mean, na.rm=TRUE)
  right <- apply(right, 1, mean, na.rm=TRUE)
  nc    <- c(left, right)
  phat  <- c(pp, pp)
  k     <- length(left)
  side  <- c(rep('left', k), rep('right', k))
  ggplot(mapping=aes(x=phat, y=nc, col=side)) + geom_line() + xlab(expression(hat(P))) + 
      ylab('Non-Coverage') + ylim(0, 0.1) + labs(caption=title)
  }
for(nam in .q(loess, linear, quadratic))
  print(plotit(R[[nam]], title=nam))

Overall non-coverage for loess :    left= 0.038  right= 0.033

Overall non-coverage for linear :   left= 0.019  right= 0.015

Overall non-coverage for quadratic :    left= 0.014  right= 0.013

The desired non-coverage probability is 0.025, which is achieved for interior predicted probabilities. The problem with coverage probabilities in the tails seems to be due to wild estimates (especially for loess) or extrapolations at more rare predicted probabilities, i.e., those closer to 0 or 1. Linear logistic calibration worked better than the more flexible calibration estimators. Note that the simulations were stacked in favor of linear calibration.

Calibration for and

Try two larger sample sizes.

set.seed(8)
Ns <- c(800, 1500)
for(N in Ns) {
  fi <- paste0('cal', N, '.rds')
  if(file.exists(fi)) Rn <- readRDS(fi) else {
    Rn        <- vector('list', 5)
    names(Rn) <- c('loess', 'linear', 'quadratic', 'basic', 'apparent')
    for(est in 1 : 5) {
      cat('\nEstimator', est, '\n')
      i <- 0
      r <- replicate(1000, sim1(N, np, B=300, smoother=sm[[est]]))  # 91m
      Rn[[est]] <- r
    }
    saveRDS(Rn, fi)
  }
  if(N ==  800) R800  <- Rn
  if(N == 1500) R1500 <- Rn
}

Again compute accuracies of bias-corrected calibration estimates.

for(N in Ns) {
  Rn <- if(N == 800) R800 else R1500
  d <- lapply(Rn, convert2dt)
  cap <- paste0('By groups of SD(Phat), n=', N)
  g <- pacc(d, stratby=expression(cut2(sphat, g=4))) + labs(caption=cap)
  print(g)
  cap <- paste0('By groups of min(tail counts), n=', N)
  g <- pacc(d, stratby=expression(cut2(count, c(20, 50, 100)))) + labs(caption=cap)
  print(g)
}


table(group)

group
[0.166,0.208) [0.208,0.218) [0.218,0.227) [0.227,0.266] 
        62500         62500         62500         62500


table(group)

group
[  0, 20) [ 20, 50) [ 50,100) [100,400] 
    27598     30189     36319    155894


table(group)

group
[0.180,0.206) [0.206,0.213) [0.213,0.220) [0.220,0.247] 
        62500         62500         62500         62500


table(group)

group
[  0, 20) [ 20, 50) [ 50,100) [100,750] 
    16879     22076     25263    185782

Now display confidence interval non-coverage proportions.

for(N in Ns) {
  Rn <- if(N == 800) R800 else R1500
  for(nam in .q(loess, linear, quadratic))
    print(plotit(Rn[[nam]], title=paste0(nam, ', N=', N)))
}

Overall non-coverage for loess, N=800 : left= 0.017  right= 0.024

Overall non-coverage for linear, N=800 :    left= 0.014  right= 0.01

Overall non-coverage for quadratic, N=800 : left= 0.02  right= 0.015

Overall non-coverage for loess, N=1500 :    left= 0.017  right= 0.019

Overall non-coverage for linear, N=1500 :   left= 0.007  right= 0.014

Overall non-coverage for quadratic, N=1500 :    left= 0.016  right= 0.013

Especially for linear and quadratic calibration, increasing the sample size made coverage larger than 0.95 for linear and quadratic calibration. Perhaps it’s best to have this conservatism so that the method will not be anti-conservative for small sample sizes.

Summary

I considered approximate confidence bands for the true model performance indexes and entire calibration curves as estimated by a bootstrap overfitting-corrected approach. The dual standard deviation approximate bootstrap confidence interval ABCLOC balances tail non-coverage probabilities by allowing the bootstrap distribution of estimated biases to be asymmetric. ABCLOC seems to provide accurate enough confidence limits for individual accuracy scores such as the Brier score, calibration slope, and to a lesser extent Somers’ , and I have a similar expectation of good performance for other scaler indexes. Considering vector measures such as whole calibration curves evaluated over a sequence of predicted risks, ABCLOC can be conservative. ABCLOC intervals are useful, and are certainly better than having no uncertainty intervals. They require no additional computation time.

I hope that others will refine ABCLOC, and run more diverse simulations.

The choice of calibration curve estimator also matters, and an important point demonstrated by simulation is the difficulty of estimating calibration (not to mention its uncertainty) at very low or high predicted probabilities. The more flexible loess method has worse confidence interval coverage, especially for predicted risks far from the mean. When the true calibration curve is logit-linear, linear calibration estimates, fitted by running binary logistic regression with a sole predictor equal to the logit of predicted risk, are superior as expected. Quadratic calibration is a good compromise, and the performance of restricted cubic spline estimators with 4 knots should also be explored. An advantage of parametric calibration methods (linear, quadratic, spline, …) is that just the parameters (3 for quadratic) of the estimated calibration curve can be saved for each bootstrap fit, and one does not need to track the individual calibration curve points (50 in number in simulations presented here) as must be done for loess.

At present, the approach of saving only the calibration parameters is only implemented in validate() functions for calibration intercept and slope.

Computing Environment

grateful::cite_packages(pkgs='Session', output='paragraph', out.dir='.',
    cite.tidyverse=FALSE, omit=c('grateful', 'ggplot2'))

We used R version 4.5.1 (R Core Team 2025) and the following R packages: data.table v. 1.17.8 (Barrett et al. 2025), Hmisc v. 5.2.4 (Harrell Jr 2025a), rms v. 8.1.0 (Harrell Jr 2025b).

The code was run on macOS Sequoia 15.6 on a Macbook Pro M2 Max.

References

Barrett, Tyson, Matt Dowle, Arun Srinivasan, Jan Gorecki, Michael Chirico, Toby Hocking, Benjamin Schwendinger, and Ivan Krylov. 2025. data.table: Extension of “data.frame”. https://doi.org/10.32614/CRAN.package.data.table.

Harrell Jr, Frank E. 2025a. Hmisc: Harrell Miscellaneous. https://hbiostat.org/R/Hmisc/.

———. 2025b. rms: Regression Modeling Strategies. https://hbiostat.org/R/rms/.

R Core Team. 2025. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. https://www.R-project.org/.

Reuse

CC BY 4.0

Minimal-Assumption Estimation of Survival Probability vs. a Continuous Variable

Frank Harrell — Sat, 19 Apr 2025 05:00:00 GMT

Background

This article considers the following setting. Suppose we have one continuous predictor and an outcome variable and we wish to estimate a smooth, usually nonlinear, relationship between and some property of such as the mean or the probability that exceeds some specified value. When there is no censoring on , one can estimate such a smooth relationship nonparametrically using a standard smoother such as loess or the R “super smoother” supsmu. Semiparametric ordinal regression, using a regression spline for is also a good approach.

Now suppose that represents the time until an event, where there may be right-censoring, i.e., for some observations the time to event is only known to be beyond the last recorded follow-up time. If we were willing to make an assumption such as proportional hazards (PH), we could easily estimate the curve in question by fitting a Cox PH model using a regression spline in . But nonparametric smoothers along the lines of loess or supsmu have not been developed for this setting, and we stiill need to estimate the relationship with minimal assumptions. For example, we may want to avoid making the PH assumption (parallelism over in for different values of ).

An especially important application is the estimation of calibration curves, which is the process of estimating how the predicted probability of surviving past a specific relates to the estimated actual probability of survival. Here the predicted survival probability is the sole continuous covariate.

Binning and computing stratified Kaplan-Meier estimates is not a competitive procedure due to noise from not utilizing interpolation leading to increased mean squared error, and from the arbitrary choice of bins.

Estimation Methods

The vs. estimation methods considered here are as follows.

hazard regression using the R polspline package’s hare function. hare uses adaptive linear splines in and in to find a smooth function. Non-parallelism (e.g., non-PH) is handled by adaptively adding product terms involving linear spline terms in and in .
adaptive ordinal regression using the R rms package’s adapt_orm function (available as of version 8.0-1) with right-censoring, starting by modeling with a restricted cubic spline function with 4 knots. Four link functions are tried: logit, probit, log-log, and complementary log-log. The link function resulting in the lowest deviance is selected, and fits using that link are then tried with 0, 3, 4, 5, and 6 knots, where 0 represents a linear fit. The fit with the best AIC is selected. This fit is then used to estimate survival probabilities at a specific over a regular grid of . The models considered span a range from PH to non-PH accelerated failure time models.
moving overlapping window Kaplan-Meier (KM) estimates at a specific using the R Hmisc package movStats function. At each distinct value occurring in the data, an interval containing eps observations on either side of is formed. For each interval, the KM estimate at is computed. By default, these are then smoothed over all the estimates at all distinct values, using the R super smoother. The amount of smoothing in supsmu is controlled by the bass parameter.

Simulation

The following simulations under one data generating mechanism estimate the performance of each of the above methods, the last method being used multiple times for different eps (and bass if smoothing KM estimates).

For sample sizes ranging from 35 to 500, simulate right-censored data generated from a log-logistic accelerated failure time model that is quadratic in . Our goal is to estimate , i.e., 3-unit survival probability as a function of the sole predictor . Define a function simdat to simulate a single dataset, with the censoring distribution being . Plot the true as a function of .

require(Hmisc)
require(rms)
require(ggplot2)
require(data.table)
require(polspline)
simdat <- function(n) {
  cens <- runif(n, 2, 6)
  x    <- runif(n)
  # Logistic AFT model
  lp   <- 2 - 2 * (x - 1) ^ 2
  t    <- exp(lp + rlogis(n) / 2)
  e    <- t <= cens
  y    <- pmin(t, cens)
  S    <- survival::Surv(y, e)
  data.table(x, y, e)
}

# Function to compute true survival probability at t=3
# T = exp(lp + r / 2)
# P(T > 3) = P(exp(lp + r / 2) > 3) = P(lp + r / 2 > log(3)) =
# P(r / 2 > log(3) - lp) = P(r > 2 * (log(3) - lp))
surv <- function(x) {
  lp <- 2 - 2 * (x - 1) ^ 2
  1 - plogis(2 * (log(3) - lp))
}
x <- seq(0, 1, by=0.01)
plot(x, surv(x), type='l', ylab='S(3 | x)')

For this setup the probability that an observation is right-censored is 0.505.

Next create a function that runs any of the estimation methods on a dataset and computes its integrated mean squared error (and its square root) in estimating the true over . Squared errors are computed for each and averaged over all 101 values.

# Function to compute sqrt of average (over grid of x) 
# mean squared error in estimating the true function of x
# using one of several methods

crmse <- function(dat, meth, msmooth='smoothed',
                  eps=15, bass=8, penalty='BIC', u=3, pl=FALSE) {
  xs <- seq(0, 1, by = 0.01)
  x  <- dat$x
  y  <- dat$y
  e  <- dat$e
  S <- survival::Surv(y, e)
  O <- Ocens(y, ifelse(e == 1, y, Inf))
  if(meth == 'hare') {
    f <- if(penalty == 'BIC') hare(y, e, x, maxdim=6)
      else hare(y, e, x, penalty=2, maxdim=6)
    s <- 1 - phare(u, xs, f)
  }
  else if(meth == 'orm') {
    f <- adapt_orm(x, O)
    opt_link <<- c(opt_link, f$family)
    opt_df   <<- c(opt_df,   f$stats['d.f.'])
    s <- survest(f, data.frame(x=xs), times=u, conf.int=0)$surv
  }
  else {
    if(nrow(dat) < 2 * eps) return(NA_real_)
    if(msmooth == 'raw')
      f <- movStats(S ~ x, times=u, msmooth='raw', eps=eps, melt=TRUE)
    else
      f <- movStats(S ~ x, times=u, msmooth=msmooth,
                    eps=eps, bass=bass, melt=TRUE)
    s <- approx(f$x, 1 - f$incidence, xout=xs, rule=2)$y
  }
  if(pl) {
    plot(xs, s, type='l', xlab='x', ylab=expression(hat(S)(t)), ylim=c(0,1))
    lines(xs, surv(xs), col='red')
    title(sub=paste(meth, msmooth, eps, bass))
  }
  c(rmse=sqrt(mean((s - surv(xs)) ^ 2)))
}

Define a function that for one dataset runs all estimation methods.

run <- function(dat, u=3, pl=FALSE) {
  # Not all parameters pertain to all methods
  # Non-applicable parameters are set to NA by rbindlist(fill=TRUE)
  u1 <- expand.grid(meth = 'hare', penalty=c('AIC', 'BIC'))
  u2 <- data.frame( meth = 'orm')
  u3 <- expand.grid(meth = 'km', msmooth='raw',
                    eps=c(10,15,20,25,30))
  u4 <- expand.grid(meth = 'km', msmooth='smoothed',
                    eps=c(10,15,20,25,30), bass=c(1,3,5,7,9))
  u <- rbindlist(list(u1, u2, u3, u4), fill=TRUE)
  g <- function(x) if(is.na(x)) '' else x
  u[, .(rmse = crmse(dat, meth, as.character(msmooth),
                     eps, bass, penalty, pl=pl),
        method=paste(meth, g(msmooth), g(eps), g(bass), g(penalty))),
    by=.(meth, msmooth, eps, bass, penalty)]
}

Now run the simulations. To gain resolution in while minimizing the number of simulations and obtaining precise results, simply draw one simulated dataset per , and later smooth the results with respect to .

w <- data.table(n = 35 : 500)
set.seed(2)
if(file.exists('sim.rds')) {
  r        <- readRDS('sim.rds')
  opt      <- readRDS('opt.rds')
  opt_link <- opt$link
  opt_df   <- opt$df
  } else {
    opt_link <- character(0)
    opt_df   <- integer(0)
    r        <- w[, run(simdat(n)), by=n]
    opt      <- list(link=opt_link, df=opt_df)
    saveRDS(r,   'sim.rds')
    saveRDS(opt, 'opt.rds')
}
cat('\nFrequencies of selected links:\n\n')


Frequencies of selected links:

table(opt_link)

opt_link
 cloglog logistic   loglog   probit 
      26      237       63      140

cat('\nFrequencies of optimum number of x parameters:\n')


Frequencies of optimum number of x parameters:

table(opt_df)

opt_df
  1   2   3   4   5 
 90 278  53  21  24

r[, eps  := factor(eps)]
r[, bass := factor(bass)]
# Mark the best performing km raw and km smoothed settings
r[, best := (meth == 'km' & eps == 30) & (
              (msmooth =='raw') |
              (msmooth == 'smoothed' & bass == 9) )]

Graph results for methods that don’t have parameters, and the performance of the other methods, at the parameters resulting in lowest root mean squared error overall. These are eps=30, indicating moving windows with 30 observations on each side of the target x value, and bass=9 indicating maximum smoothing using the “super smoother” R function supsmu.

ggplot(r[meth %in% c('hare', 'orm') | best,],
       aes(x=n, y=rmse, col=method)) +
  geom_smooth(se=FALSE) +
  labs(caption='hare and orm vs. best moving window KM estimators')

Oddly enough, hare BIC performed slightly better than AIC for low sample sizes, and there was no advantage of BIC for large . Moving-window Kaplan-Meier estimates, either smoothed or unsmoothed, did not perform as well as the other methods. The best integrated mean squared estimation error was had with the adaptive-link orm method.

Next graph results for the method that has one parameter, unsmoothed moving-window KM.

ggplot(r[meth == 'km' & msmooth == 'raw',],
       aes(x=n, y=rmse, col=eps)) +
  geom_smooth(se=FALSE) +
  labs(caption='Unsmoothed KM estimator performance by sample size on either side of target x')

Best overall mean squared estimation error resulted from eps=30 observations on either side of the target predictor value x.

Now consider the method with two parameters – smoothed moving-window KM.

ggplot(r[meth == 'km' & msmooth == 'smoothed',],
       aes(x=n, y=rmse, col=eps)) +
  geom_smooth(se=FALSE) + facet_wrap(~ bass) +
  labs(caption='Smoothed KM performance by eps and smoothing parameter')

Show this another way to better judge effect of the smoothing parameter bass.

ggplot(r[meth == 'km' & msmooth == 'smoothed',],
       aes(x=n, y=rmse, col=bass)) +
  geom_smooth(se=FALSE) + facet_wrap(~ eps) +
  labs(caption='Smoothed KM performance by eps and smoothing parameter')

The best bass value is 9, i.e., maximum smoothing. The best window size was eps=30.

Recommendations

The overall winner is considering four link functions in an ordinal regression model with right censoring, fitting the predictor x with a restricted cubic spline function, and choosing the link with minimum deviance. The number of knots in the spline (or linearity) can be selected using AIC. If one wanted to accommodate more exotic effects of x over time, e.g., complex non-parallelism in survival curves, hare is a good choice.

Computing Environment

grateful::cite_packages(pkgs='Session', output='paragraph', out.dir='.',
    cite.tidyverse=FALSE, omit=c('grateful', 'ggplot2'))

We used R version 4.4.2 (R Core Team 2024) and the following R packages: data.table v. 1.17.0 (Barrett et al. 2025), Hmisc v. 5.2.4 (Harrell Jr 2025a), polspline v. 1.1.25 (Kooperberg 2024), rms v. 8.0.1 (Harrell Jr 2025b).

The code was run on macOS Sequoia 15.4.1 on a Macbook Pro M2 Max.

References

Harrell Jr, Frank E. 2025a. Hmisc: Harrell Miscellaneous. https://hbiostat.org/R/Hmisc/.

———. 2025b. rms: Regression Modeling Strategies. https://hbiostat.org/R/rms/.

Kooperberg, Charles. 2024. polspline: Polynomial Spline Routines. https://CRAN.R-project.org/package=polspline.

R Core Team. 2024. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. https://www.R-project.org/.

Reuse

CC BY 4.0

Statistical Computing Approaches to Maximum Likelihood Estimation

Frank Harrell — Thu, 28 Nov 2024 06:00:00 GMT

Overview

Maximum likelihood estimation (MLE) is a gold standard estimation procedure in non-Bayesian statistics, and the likelihood function is central to Bayesian statistics (even though it is not maximized in the Bayesian paradigm). MLE may be unpenalized (the standard approach) or various penalty functions such as L1 (lasso, absolute value penalty), and L2 (ridge regression; quadratic) penalties may be added to the log-likelihood to achieve shrinkage (aka regularization). I have been doing MLE my entire career, using mainly a Newton-Raphson algorithm with step-halving to achieve rapid convergence. I never stopped to think about other optimization algorithms, and the R language has a good many excellent algorithms included in the base stats package. There is also an excellent maxLik R package devoted to MLE (not used here), and two of its vignettes provide excellent introductions to MLE. General MLE background and methods including more about penalization may be found here, which includes details about how to do QR factorization and to back-transform after the fit.

If you think that ROC area or classification accuracy are good objective functions/optimality criteria, please think again.

In this article I’ll explore several optimization strategies and variations of them, in the context of binary and ordinal logistic regression. The programming scheme that is used here is the one used by many R packages: Write functions to compute the scaler objective function (here, -2 log likelihood), the gradient vector (vector of first derivatives), and the hessian matrix (matrix of second derivatives), and pass those functions to general optimization functions. Convergence (at least to local minima for -2 LL (log of the likelihood function)) is achieved for smooth likelihood functions when the gradient vector values are all within a small tolerance (say ) of zero or when the -2 LL objective function completely settles down in, say, the significant digit. The gradient vector is all zeros if the parameter values are exactly the MLEs (local minima achieved on -2 LL).

A different strategy is to use the Bayesian system Stan by specifying LL and letting Stan analytically compute the gradient, then using the Stan optimizer to compute MLEs. If you specify priors, the optimizer provides penalized MLEs. The likelihood function is the bridge between Bayesian and frequentist methods.

This article is based on rms version 7.0-0, a major new release of the package, which will likely be available on CRAN around 2025-01-08.

History

The R rms package lrm function is dedicated to maximum likelihood estimation (MLE) for fitting binary and ordinal (proportional odds) logistic regression models using the logit link, with or without quadratic (ridge) penalization. Semiparamteric regression models, also called ordinal regression models, allow one to do efficient analyses without depending on how the dependent variable Y is transformed. Ordinal models encode the entire cumulative distribution function of Y by having an intercept for each distinct Y level less one. For ordinal models, versions of lrm before rms 6.9-0 were efficient for up to 400 distinct Y-values (399 intercepts) in the sense that execution time was under 10 seconds for 10,000 observations on 10 predictors. The rms orm function is intended for modeling continuous outcome Y variables and was efficient for up to 8000 intercepts prior to rms 7.0-0. orm implements 4 link functions other than the logit. For rms 7.0-0, lrm and orm run in 2.5s for a sample size of 300,000 with continuous Y and 20 predictors, i.e., with 299,999 intercepts (9.5s for 40 predictors). lrm uses the R function lrm.fit for its heavy lifting, and likewise orm uses orm.fit. Much of lrm.fit was written in 1980 and served as the computational engine for the first SAS procedure for logistic regression, PROC LOGIST. It used Fortran 77 for computationally intensive work, and used only a Newton-Raphson algorithm with step-halving for iterative MLE. On rare occasions when serious collinearities were present, such as when multiple continuous variables were fitted using restricted cubic splines (which use a truncated power basis), lrm.fit would fail to converge.

Are Intercepts Regular Parameters?

Let be a smooth cumulative distribution function. The cumulative probability class of ordinal semiparametric models can be written as follows. It is more traditional to state the model in terms of but we use so that higher predicted values are associated with higher , and when is the logistic distribution the ordinal logistic (proportional odds) model reduces exactly to a binary logistic model. Let the ordered distinct values of be denoted by and let the intercepts associated with be , where because . Let . Then the cumulative probability semiparametric model is

When is the extreme value type I (also called the Gumbel minimum value) distribution , the inverse function is the link and the model is a proportional hazards (PH) model. The Cox 1972 PH model uses the only generating distribution such that the marginal (getting rid of the s) distribution of the ranks of can be evaluated without multi-dimensional integrals. This gives rise to Cox’s partial likelihood, which until time-dependent covariates are included can be computationally fast for any number of distinct failure times. The Cox approach requires a second step to estimate the intercepts (underlying survival curve for a person with some reference value of ). There is some arbitrariness to which second-step estimator is used, e.g., the Breslow estimator or the Kalbfleisch-Prentice estimator. And variances of are complicated because uncertainties from both of the steps must be included.

In fact, a partial likelihood, which also makes it difficult to handle interval censoring, is only needed until you realize that

the maximum likelihood estimate (MLE) of the vector of intercepts when are just the link function of all the one minus cumulative probabilities in the absence of censoring; this gives rise to instantly-computed and convergence-accelerating initial values for iterative MLE estimation
the s are always in descending order (ascending if using the more popular statement of ordinal models)
the gradient (first derivative) for the log likelihood function can be computed quickly no matter how large is
the hessian (second derivatives of the log likelihood function) can be computed in just over twice the time needed to compute the gradient, and takes twice the amount of array storage size as the gradient, so MLE scales wonderfully for large
for most needs, the entire information matrix (negative hessian) never needs to be inverted; portions of the inverse of the whole can be quickly computed without inverting the whole information matrix
the R Matrix package is made for efficient storage and calculation on such sparse hessians

Twice because the intercept portion of the hessian is tri-band diagonal and one only needs to store the diagonal and above-diagonal elements due to symmetry.

Because of the strict ordering of , MLE iterations are fast and the effective degrees of freedom of the model are more like where is the number of s. The comes from the following line of reasoning. In the no-covariate case, consider confidence bands for the empirical cumulative distribution function (ECDF) for , then fit a 4-parameter parametric distribution to the raw values. Confidence intervals for from this parametric fit will be about the same widths as those from the ECDF.

ECDF = when This exercise would also point out the lack of value in fitting flexible parametric distributions compared to generalizing the ECDF to handle covariates by using a semiparametric model.

Re-Write of `lrm.fit` and `orm.fit`

To modernize the Fortran code, better condition the design matrix ( observations and columns), and to explore a variety of optimization algorithms, I did a complete re-write of the rms package lrm and lrm.fit functions in November, 2024, for rms version 6.9-0. To reduce optimization divergence when there are extreme collinearities, and to better scale , I was interested in implementing mean centering followed by a QR factorization of to orthogonalize its columns. Details about how this is done and how the parameter estimates and covariance matrix are converted back to the original space may be found here. QR can be turned on by setting the lrm.fit argument transx to TRUE. When QR is in play, the rotated columns of are scaled to have standard deviation 1.0.

Besides dealing with fundamental statistical computing issues, I changed lrm.fit to use Therneau’s survival package concordancefit function to compute concordance probabilities used by various rank correlation indexes such as Somers’ . This got rid of a good deal of code. Previously, lrm.fit binned predicted probabilities into 501 bins to calculate rank measures almost instantly. But I decided it was time to use exact calculations now that concordancefit is so fast. Though not used in rms, concordancefit also computes accurate standard errors.

A new Fortran 2018 subroutine lrmll was written to efficiently calculate the -2 log-likelihood function (the deviance), the gradient vector, and the hessian matrix of all second partial derivatives of the log-likelihood with respect to intercept(s) and regression coefficients (slopes) .

lrm.fit now implements several optimization algorithms. When Y is binary and there is no penalization, it has the option to use glm.fit(..., family=binomial()) which runs iteratively reweighted least squares, a fast-converging algorithm for binary logistic regression (but does not extend to ordinal regression). lrm.fit implements 8 optimization algorithms.

optional for proportional odds models (according to lrm.fit initglm argument), does a first pass with glm.fit that fits a binary logistic regression for the probability that Y is greater than or equal to the median Y; this fit can then be used for starting values after shifting the default starting intercept values so that the middle intercept matches the intercept from the binary fit
nlminb: a function in the stats package, tied with NR as the fastest algorithm in general; uses hessians. This uses Fortran routines in the Bell Labs port library for which the original paper may be found here.
NR: Newton-Raphson iteration with step-halving, implemented here as R function newtonr. This algorithm is the default because it has the advantage of having full control over convergence criteria. It requires convergence with respect to 3 simultaneous criteria: changes in -2 LL, changes in parameter estimates, and nearness of gradient to zero. The user can relax any of the 3 criteria thresholds to relax conditions. This is the optimization method used in the old lrm.fit function, but written in Fortran there for slightly increased speed. Defaults for tolerance parameters are such that eps (iteration-to-iteration change in -2 LL) will usually dictate when convergence called.
LM: Levenberg-Marquart algorithm, which is a kind of Newton method with generalized step-halving
glm.fit: for binary Y without penalization only
nlm: the stats function that is usually recommended for maximum likelihood, but I found it is slower than nlminb without offering other advantages
BFGS and L-BFGS-B using the stats optim function: fast general-purpose algorithms that do not require the hessian, so these can be used with an unlimited number of intercepts as long as the user sets the lrm.fit parameter compvar to FALSE so that the hessian is not calculated once after convergence
CG and Nelder-Mead: see optim

This was inspired by the MASS package polr function

The last four methods do not involve computing the hessian, which is the most computationally intensive calculation in MLE. But they are still slower overall if you want to get absolute convergence, due to requiring many more evaluations of the object function (-2 LL).

See this for useful comparisons of some of the algorithms.

For rms 7.0-0, lrm.fit and orm.fit were changed to use the R Matrix package for much more efficient handling of sparse Hession/information matrices. Now there are no limitations on the number of distinct Y-values analyzed by lrm and orm. The primary differences between the two modeling procedures are

lrm only implements a single link function (logistic)
lrm implements multiple optimization methods
orm only implements Newton-Raphson optimization (with step-halving) and Levenberg-Marquardt
lrm output (from print.lrm) includes rank correlation model performance indexes that are more suitable for discrete Y
orm output (from print.orm) includes only Spearman’s as a rank predictive discrimination measure; this is more suitable for continuous Y
orm has a Quantile and an ExProb method; both lrm and orm have Mean methods, as means work on discrete numeric Y (unlike quantiles)
lrm implements transx for QR orthonormalization of the design matrix
orm implements scale for mean-centering and standard deviation scaling of

Background: Convergence

The two most commonly used convergence criteria for MLE are

relative convergence: stop iterations when the change in deviance is small or when the relative change in parameter estimates is small
absolute convergence: stop iterations when the first derivative of the log likelihood is small in absolute value for all parameters, or there is a very small absolute change in parameter values

Absolute convergence with respect to the first derivatives (gradients) is similar to demanding that none of the regression parameters change very much since the last iteration. From the standpoint of what is important statistically, convergence of the deviance (what the gold standard likelihood ratio test is based on) is sufficient with respect to what matters. Changing parameter values when the deviance does not change in the decimal place will be buried in the noise. Absolute convergence may affect , but relative convergence tends to result in a very stable . You might deem convergence satisfactory when parameter estimates between successive iterations change by less than 0.05 standard errors (which would require evaluation of the hessian to know), but approximately this corresponds to relative convergence judged by the deviance.

Despite achieving statistically relevant convergence more easily, there is a real issue of reproducibility. Different algorithms and software may result in semi-meaningful differences in taken in isolation (ignoring how much of is noise). Only by having all the implementations achieve absolute convergence will different analysts be very likely to reproduce each others’ work. If this is important to you, either avoid the BFGS algorithms (for which the R optim function does not have an absolute convergence criterion) or use a highly stringent relative convergence criterion, e.g., specify the lrm.fit argument reltol as . Below I explore how convergence and execution time are affected by reltol.

Overview of Findings

opt_method='NR' and 'nlminb' are the fastest methods, even slightly faster than glm.fit.

Limited tests of transx in lrm.fit to use QR factorization does not show not much benefit, but see below for details.

For ordinal Y, using opt_method='BFGS' with compstats=FALSE for lrm mimics the polr function in the MASS package. For a large number of intercepts, lrm.fit is much faster due to computing the deviance and derivatives in highly efficient compiled Fortran and capitalizing on sparse matrices.

Setting initglm=TRUE tells lrm.fit to get initial ordinal model parameter values from a binary logistic glm.fit run when cutting Y at the median. This does not seem to offer much benefit over setting starting values to covariate-less MLEs of (which are calculated instantly when ) and setting . In one example using nlminb the algorithm actually diverged with initglm=TRUE but ran fine without it. This is probably due to large intercept values with many distinct Y values.

Extensive tests of opt_method='BFGS' show good stochastic performance (convergence to what matters, deviance-wise), but unimpressive execution time if you want absolute convergence by setting reltol to a small number.

The best overall algorithm that uses the hessian is NR in terms of speed and convergence, with nlminb and LM close seconds. NR is the method used in the old lrm.fit, so for most datasets, the new optimization options are not needed.

Even though lrm.fit is optimized for the logistic link function, there is not much difference in execution time between lrm.fit and orm.fit for binary and ordinal logistic models.

Validation

Two kinds of validations appear below.

Validation of the Fortran-calculated deviance and derivatives: for a given small dataset, get parameter estimates from rms::orm with tight convergence criteria, and test the Fortran code by evaluating the deviance and derivatives at these parameter values. The gradient (first derivative) should be very close to zero, and the deviance should be identical to that from orm. The inverse of the negative of the hessian matrix should equal the variance-covariance matrix computed by orm.
Validate lrm.fit overall by letting it pick its usual starting values for iteration, and compare its output to that from orm and other fitting functions including the last pre-6.9-0 version of lrm.fit which is named olrm below.

require(rms)
# Fetch the old lrm.fit (function olrm)
require(orms)

# Define simple timer functions with and without printing

stime <- function(...) {
  ti <- system.time(...)['elapsed']
  cat('Elapsed time: ', ti, 's\n', sep='')
  invisible(ti)
}

stim <- function(...) system.time(...)['elapsed']

# Define a function that will time a series of lines of code, repeating each
# reps times (default is 10).  When only one line of code is given, the
# elapsed execution time is printed and the result of the line is returned.
# Otherwise, a vector of run times corresponding to all the lines of code is
# returned, and the values returned from the lines are stored in the global
# environment in object Res, with components named according to inputs to tim.
# When reps > 1 the values from running code lines are from the last rep.

tim <- function(...) {
  reps <- 10
  w <- sys.call()
  n <- names(w)
  m <- length(n)
  k <- 2 : m
  if('reps' %in% n) {
    i    <- which(n == 'reps')
    reps <- eval(w[[i]])
    k    <- setdiff(k, i)
  }
  n <- n[k]
  m <- length(n)
  r <- numeric(m)
  Res        <<- vector('list', m)   # put in global environment
  names(r)   <- n
  names(Res) <<- n
  l <- 0
  for(i in k) {
    l <- l + 1
    s <- stim(for(j in 1 : reps)
    R <- eval(w[[i]], parent.frame())) / reps
    r[l]      <- s
    Res[[l]] <<- R
  }
  label(r) <- paste('Per-run execution time in seconds, averaged over', reps, 'runs')
  if(m == 1) {
    print(r)
    return(R)
  }
  r
}

m   <- function(x) max(abs(x))
mad <- function(a, b) c(mad    =     mean(abs(a - b)),
                        relmad = 2 * mean(abs(a - b) / (abs(a) + abs(b))))
wratio <- function(r) exp(max(abs(log(r))))  # worst ratio, whether < or > 1.0

# Function creating a table of matrix dimensions for matrices in a list
mdim <- function(w) {
  i <- sapply(w, is.matrix)
  g <- function(x) c(rows=nrow(x), columns=ncol(x))
  sapply(w[i], g)
}
            
# Function to summarize a series of model fits stored in Res
smod <- function() {
  max_abs_u <- sapply(Res, function(x) if(length(x$u)) m(x$u) else NA)
  iter      <- sapply(Res, function(x) if(length(x$iter)) tail(x$iter, 1) else NA)
  deviance  <- sapply(Res, function(x) if(length(x$deviance)) tail(x$deviance, 1) else NA)
  print(data.frame(deviance, max_abs_u, iter))
  l <- length(Res)
  n <- names(Res)
  if(l > 1) {
    for(i in 1 : l) {
      r <- Res[[i]]
      # See if polr or BFGS
      a <- inherits(r, 'polr') || (length(r$opt_method) && r$opt_method=='BFGS')
      r$var <- if(inherits(r, 'orm'))
        vcov(r, intercepts='all') else if(! a) vcov(r)
      if(inherits(r, 'polr'))
        r$coefficients <- c(-r$zeta, coef(r))
      Res[[i]] <- r
    }
    
    cat('\nMaximum |difference in coefficients|,',
        'Maximum |relative difference|\n',
        'worst ratio of covariance matrices\n\n')
    d <- NULL
    for(i in 1 : (l-1))
      for(j in (i+1) : l) {
        ri <- Res[[i]]; rj <- Res[[j]]
        comp <- paste(n[i], 'vs.', n[j])
        co <- mad(ri$coefficients, rj$coefficients)
        w <- data.frame(Comparison         = comp,
                        `Max |difference|` = co[1],
                        `Max |rel diff|`   = co[2],
                        `Cov ratio`        = NA, check.names=FALSE)
        if(length(ri$var) && length(rj$var)) 
          w$`Cov ratio` <- wratio(ri$var / rj$var)
        d <- rbind(d, w)
        }
  }
  d$`Cov ratio` <- ifelse(is.na(d$`Cov ratio`), '', format(d$`Cov ratio`))
  rownames(d) <- NULL
  d
}

Check -2 Log Likelihood and Derivatives for a Simple Model

First define an R function that makes it easy to run the Fortran subroutine.

rfort <- function(x, y, alpha, beta, what=2L, debug=0L, penhess=1L,
                  offset=rep(0., n), wt=rep(1., n),
                  penmat=matrix(0., p, p)) {
  x  <- as.matrix(x)
  p  <- ncol(x)
  n  <- nrow(x)
  yd <- sort(unique(y))
  k  <- max(yd)
  nv <- as.integer(k + p)
  if(length(yd) != k + 1 || any(yd != 0 : k))
    stop('y must be coded 0-k for lrmll')
  storage.mode(x) <- storage.mode(offset) <- storage.mode(wt) <- storage.mode(penmat) <-
    storage.mode(alpha) <- storage.mode(beta) <- 'double'

  w <- .Fortran('lrmll', n, as.integer(k), p, x, as.integer(y),
                 offset, wt, penmat, alpha, beta,
                 logL=numeric(1), grad=numeric(nv),
                 a=matrix(0e0, k, 2), b=matrix(0e0, p, p), ab=matrix(0e0, k, p),
                 what=as.integer(what), debug=as.integer(debug),
                 penhess=as.integer(penhess), salloc=integer(1))
  # lrmll creates 3 compact hessian submatrices
  # Put them together into a single hessian
  w$hess <- infoMxop(w[c('a', 'b', 'ab')])
  w
}

Binary Y

x <- 1 : 10
y <- c(0, 1, 0, 0, 0, 1, 0, 1, 1, 1)

# From orm.  Deviance = 13.86294, 10.86673
alpha <- -2.4412879506377 ; beta <- 0.4438705364796 
w <- rfort(x, y, alpha, beta)
w$logL

[1] 10.86673

w$grad

[1] -1.814104e-13 -1.156408e-12

w <- rfort(x, y, alpha, beta, what=3L)
w$hess   # negative inverse of covariance matrix

          [,1]       [,2]
[1,] -1.813852  -9.976185
[2,] -9.976185 -66.123262

- solve(vcov(glm(y ~x, family=binomial(), control=list(epsilon=1e-12))))

            (Intercept)          x
(Intercept)   -1.813852  -9.976185
x             -9.976185 -66.123262

Res <- list(glm     = glm(y ~ x, family=binomial(), control=list(epsilon=1e-12)),
            olrm    = olrm(x, y, eps=1e-10),
            lrm.fit = lrm.fit(x, y, reltol=1e-12),
            orm     = orm(y ~ x, eps=1e-10) )

smod()

        deviance    max_abs_u iter
glm     10.86673           NA    4
olrm    10.86673 2.220446e-15   NA
lrm.fit 10.86673 3.270271e-08    5
orm     10.86673 8.326673e-17    5

Maximum |difference in coefficients|, Maximum |relative difference|
 worst ratio of covariance matrices

        Comparison Max |difference| Max |rel diff| Cov ratio
1     glm vs. olrm     5.551115e-16   4.320309e-16         1
2  glm vs. lrm.fit     5.551115e-16   4.320309e-16         1
3      glm vs. orm     5.273559e-16   3.695001e-16         1
4 olrm vs. lrm.fit     0.000000e+00   0.000000e+00         1
5     olrm vs. orm     2.775558e-17   6.253079e-17         1
6  lrm.fit vs. orm     2.775558e-17   6.253079e-17         1

Cov ratio in the above output is the anti-log of the absolute value of the log of the ratio of elements of two covariance matrices. So it represents the worst disagreement in the two matrices, with 1.0 being perfect agreement to 7 decimal places. Max |difference| is the highest absolute difference in estimated regression coefficients between two methods, and Max |rel diff| is the maximum ratio of absolute differences to the sum of absolute values of two coefficient estimates.

Y=0, 1, 2

x <- 1 : 10
y <- c(0, 2, 0, 1, 0, 2, 2, 1, 1, 2)
f <- orm(y ~ x, eps=1e-10) # deviance 21.77800  19.79933
- solve(vcov(f, intercepts='all'))

          y>=1       y>=2           x
y>=1 -2.336337   1.148893   -5.103426
y>=2  1.148893  -2.657167  -10.455125
x    -5.103426 -10.455125 -110.128337

- solve(vcov(lrm(y ~ x, eps=1e-10)))

          y>=1       y>=2           x
y>=1 -2.336337   1.148893   -5.103426
y>=2  1.148893  -2.657167  -10.455125
x    -5.103426 -10.455125 -110.128337

- solve(olrm(x, y, eps=1e-10)$var)

          y>=1       y>=2        x[1]
y>=1 -2.336337   1.148893   -5.103426
y>=2  1.148893  -2.657167  -10.455125
x[1] -5.103426 -10.455125 -110.128337

- solve(vcov(MASS::polr(factor(y) ~ x)))

             x       0|1       1|2
x   -110.12802  5.103400 10.455131
0|1    5.10340 -2.336318  1.148877
1|2   10.45513  1.148877 -2.657156

# Note a problem with VGAM
- solve(vcov(VGAM::vglm(y ~ x, VGAM::cumulative(reverse=TRUE, parallel=TRUE))))

              (Intercept):1 (Intercept):2           x
(Intercept):1     -2.434082      1.155134   -5.148614
(Intercept):2      1.155134     -2.580855   -9.505256
x                 -5.148614     -9.505256 -100.063658

- solve(vcov(ordinal::clm(factor(y) ~ x)))

          0|1       1|2           x
0|1 -2.336337  1.148893    5.103426
1|2  1.148893 -2.657167   10.455125
x    5.103426 10.455125 -110.128337

alpha <- c(-0.8263498291155, -2.3040967379853)
beta <- 0.3091154153068 

# Analytically compute 2nd derivative of log L wrt beta
info <- function(x) {
  p1 <- plogis(alpha[1] + x * beta) 
  p2 <- plogis(alpha[2] + x * beta)
  d <- p1 - p2
  v1 <- p1 * (1 - p1)
  v2 <- p2 * (1 - p2)
  v1 - v2
  w1 <- p1 * (1 - p1) * (1 - 2 * p1)
  w2 <- p2 * (1 - p2) * (1 - 2 * p2)
  w1 - w2
  x * x * ((w1 - w2) * d - (v1 - v2)^2) / d / d
}

# Compute 2nd derivative of log(p1 - p2) wrt beta numerically
dif <- function(x, beta) log(plogis(alpha[1] + x * beta) - plogis(alpha[2] + x * beta))
del = 1e-6
d2 <- function(x) ((dif(x, beta + del) - dif(x, beta)) / del - (dif(x, beta) - dif(x, beta - del)) / del) / del
c(info(4), info(8), info(9))

[1]  -6.88269 -24.55641 -27.93077

num <- c(d2(4), d2(8), d2(9))
print(num)

[1]  -6.882495 -24.556135 -27.931213

sum(num)

[1] -59.36984

w <- rfort(x, y, alpha, beta)
w$logL

[1] 19.79933

w$grad

[1] -1.896261e-13 -1.408873e-13 -2.330580e-12

w <- rfort(x, y, alpha, beta, what=3L)
w$hess

3 x 3 sparse Matrix of class "dgCMatrix"
                                     
[1,] -2.336337   1.148893   -5.103426
[2,]  1.148893  -2.657167  -10.455125
[3,] -5.103426 -10.455125 -110.128337

Simple Ordinal Model With Weights, Offsets, and Penalties

We first ignore weights, offsets, and penalties, then incorporate them.

set.seed(1)
x1 <- rnorm(50)
x2 <- rnorm(50)
X  <- cbind(x1, x2)
y  <- sample(0:5, 50, TRUE)
wt <- runif(50)
wt <- wt / sum(wt)
of <- rnorm(50)
pm <- cbind(c(1.2, 0.6), c(0.6, 1.2))

f <- olrm(X, y, eps=1e-15)
f$deviance

[1] 177.1350 176.8656

cof <- coef(f)
w <- rfort(X, y, cof[1:5], cof[6:7])
w$logL

[1] 176.8656

w$grad

[1]  4.160439e-11  1.768230e-11 -4.204526e-11 -7.764234e-12 -3.765321e-13
[6] -1.289524e-12  7.128742e-13

w <- rfort(X, y, cof[1:5], cof[6:7], what=3L)
range(w$hess + solve(vcov(f)))

[1] -58.68270  51.98111

# Needed reltol=1e-15 to get gradient to 1e-8 with BFGS
# CG achieved 1e-6 with default, with 475 function evaluations

g <- lrm.fit(X, y, trace=1, opt_method='nlm', gradtol=1e-14,
             transx=TRUE, compstats=FALSE)

iteration = 0
Step:
[1] 0 0 0 0 0 0 0
Parameter:
[1]  1.51634749  0.57536414 -0.08004271 -0.84729786 -2.19722458  0.00000000
[7]  0.00000000
Function Value
[1] 177.135
Gradient:
[1]  1.199041e-14 -2.664535e-15 -1.243450e-14  7.993606e-15  3.330669e-15
[6]  2.510851e+00  3.400018e+00

iteration = 4
Parameter:
[1]  1.53972787  0.60066749 -0.06107634 -0.83760533 -2.19876783 -0.07124309
[7] -0.10606549
Function Value
[1] 176.8656
Gradient:
[1] -8.387024e-11 -3.563017e-11  8.474110e-11  1.566214e-11  7.490675e-13
[6]  5.339063e-12  9.486856e-13

Last global step failed to locate a point lower than x.
Either x is an approximate local minimum of the function,
the function is too non-linear for this algorithm,
or steptol is too large.

m(g$u); g$iter  # L-BFGS-B tool factor as low as 1e2 to get u=3e-6, 50 iter

[1] 4.237055e-11

[1] 4

mad(cof, g$coefficients)

         mad       relmad 
1.066508e-14 3.650349e-14

range(f$var / vcov(g))

[1] -95.44248  25.29241

Now use weights, offset, and penalties.

f <- olrm(X, y, offset=of, weights=wt, penalty.matrix=2*pm, eps=1e-12)
f$deviance

[1] 3.475801 3.803296 3.799336

cof <- coef(f)
w <- rfort(X, y, cof[1:5], cof[6:7], offset=of, wt=wt, penmat=2*pm)
w$logL

[1] 3.799336

w$grad

[1]  4.683753e-17 -9.714451e-17 -5.204170e-18  3.469447e-18 -3.903128e-17
[6] -3.469447e-18  0.000000e+00

w <- rfort(X, y, cof[1:5], cof[6:7], offset=of, wt=wt, penmat=2*pm, what=3L)
range(w$hess + solve(vcov(f)))

[1] -1.776357e-15  1.332268e-15

g <- lrm.fit(X, y, trace=3, offset=of, weights=wt,
             penalty.matrix=2e0*pm, opt_method='nlminb', compstats=FALSE)

  0:     3.8301785:  1.23911 0.611204 -0.0435624 -0.772679 -2.45652
  3:     3.8032957:  1.50965 0.925273 0.280639 -0.513375 -2.39790
  0:     3.8032957:  1.50965 0.925273 0.280639 -0.513375 -2.39790  0.00000  0.00000
  3:     3.7993356:  1.50984 0.927634 0.283085 -0.513316 -2.39750 0.0149656 -0.0431891

m(g$u); g$iter

[1] 3.404934e-09

          iterations evaluations.function evaluations.gradient 
                   3                    4                    3

mad(coef(f), coef(g))

         mad       relmad 
7.703453e-09 7.127106e-09

g$iter

          iterations evaluations.function evaluations.gradient 
                   3                    4                    3

g$u

         y>=1          y>=2          y>=3          y>=4          y>=5 
 3.404934e-09  6.040723e-10 -1.193834e-09 -1.123210e-09 -1.671228e-09 
           x1            x2 
 3.391934e-10  7.970917e-11

f$deviance

[1] 3.475801 3.803296 3.799336

g$deviance

[1] 3.475801 3.803296 3.799336

range(vcov(f) / vcov(g))

[1] 0.9999999 1.0000002

Check Accuracy Against Old `lrm.fit` For a Variety of Levels of Y

set.seed(1)
n <- 150
w <- NULL
for(i in 1 : 40) {
  k <- sample(1 : 20, 1, TRUE)
  y <- sample(0 : k, n, TRUE)
  x1 <- runif(n)
  x2 <- exp(rnorm(n))
  X  <- cbind(x1, x2)
  f <- olrm(X, y, eps=1e-10)
  g <- lrm.fit(    X, y, opt_method='nlminb', compstats=FALSE)
  d <- coef(f) - coef(g)
  r <- wratio(vcov(f) / vcov(g))
  w <- rbind(w, data.frame(i, k, mad.beta=m(d), Cov.ratio=r))
}
range(w$Cov.ratio)

[1] 1 1

with(w, plot(k, mad.beta, log='y'))

Study Convergence and Timings

Fortran vs. R

Regarding execution speed, a key question is whether it’s worth the effort to code part of the calculations in a compiled language such as Fortran, C, or C++, as compared to just using R. Let’s explore this by coding the gradient vector calculation in R and timing it against the new Fortran code. Also write a function making the Fortran routine easy to call when computing the gradient.

The code below makes use of the facts that and . The philosophy of this code is that nothing is calculated unless it is relevant, which makes for more lines of code. For example, the code does not create extra intercepts to yield probabilities of 0 or 1, but instead handles each case Y=0, Y=, separately. For the non-interior levels of Y the gradient is very simple as probabilities are expits and not differences.

grad <- function(alpha, beta, x, y) {
  k  <- length(alpha)
  p  <- length(beta)
  f  <- plogis
  xb <- as.vector(x %*% matrix(beta, nrow=p))
  xb <- as.vector(x %*% beta)
  P1 <- P2 <- numeric(n)
  i0 <- y == 0
  ik <- y == k
  ib <- y > 0 & y < k
  P1[i0] <- 1e0
  P2[i0] <- f(alpha[1] + xb[i0])
  P1[ik] <- f(alpha[k] + xb[ik])
  P2[ik] <- 0e0
  P1[ib] <- f(alpha[y[ib]    ] + xb[ib])
  P2[ib] <- f(alpha[y[ib] + 1] + xb[ib])
  pq1    <- P1 * (1e0 - P1)
  pq2    <- P2 * (1e0 - P2)
  P      <- P1 - P2
  U      <- rep(0e0, k + p)
  
  U[1] <- - sum(1e0 - P[i0])
  U[k] <-   U[k] + sum(1e0 - P[ik])
  
  # Gradiant for intercepts
  if(k > 1) for(m in 1 : k) {  # only interior y values create complexity
    if(m < k) U[m] <- U[m] + sum(pq1[y == m] / P[y == m])
    if(m > 1) U[m] <- U[m] - sum(pq2[y == m - 1] / P[y == m - 1])
  }
  # Gradient for slopes
  for(m in 1 : p) {
    U[k + m] <-            - sum(x[i0, m] * (1e0 - P[i0]))
    U[k + m] <-   U[k + m] + sum(x[ik, m] * (1e0 - P[ik]))
    if(k > 1) for(i in 1 : (k - 1)) {
    j <- y == i
    U[k + m] <-   U[k + m] + sum(x[j,  m] * (pq1[j] - pq2[j]) / P[j])
    }
  }
  U
}

fgrad <- function(alpha, beta, x, y) rfort(x, y, alpha, beta)$grad
  
# This calls a version of the Fortran code using the alpha extension approach
fgrad2 <- function(alpha, beta, x, y) {
  x <- as.matrix(x)
  n <- nrow(x)
  p <- ncol(x)
  k <- max(y)  # y assumed to be coded 0-k
  .Fortran('lrmll2',
           as.integer(n), as.integer(k), as.integer(p),
           x, y, rep(0e0, n), rep(1e0, n), matrix(0e0, nrow=p, ncol=p),
           alpha, beta, numeric(1), u=numeric(k + p), numeric(1), 2L, 0L, 0L)$u
}

But is the elegance of following the letter of the proportional odds model’s definition worth the trouble? What if we used the trick that is most often used in MLE and Bayesian modeling where we define extra intercepts so that all values of Y appear to be interior values and the same difference in probabilities can be computed everywhere, and we did not use special cases to compute derivatives of log likelihood components? Specifically we can write the model as the following, with y = and .

by expanding the original vector of intercepts by adding and . Then for any . The is chosen so that is indistinguishable from 1 and 0.

We need to run some computational tests to make sure that the upcoming shortcuts do not cause any computational inefficiencies or inaccuracies.

# Check that R computes expit very quickly for extreme values of x
tim(smallvals = plogis(rep(  1, 100000)),
    m50       = plogis(rep(-50, 100000)),
    p50       = plogis(rep( 50, 100000)))

Per-run execution time in seconds, averaged over 10 runs 
smallvals       m50       p50 
   0.0011    0.0009    0.0009

# Check that taking the log of probabilities is as accurate as
# using plogis' special log probability calculation
x <- seq(-50, 50, by=1); m(log(plogis(x)) - plogis(x, log=TRUE))

[1] 8.881784e-16

So it appears that the “intercept extension” approach will not cause any numerical problems. To code this method while computing the gradient, we need the derivative of the log of the difference in probabilities (call this ) given above. Consider a general parameter which may be one of the (interior) s or one of the s.

Since the main part of ,

is the 0/1 indicator function when and is when . Now code this in R.

grad2 <- function(alpha, beta, x, y) {
  k     <- length(alpha)
  p     <- length(beta)
  xb    <- as.vector(x %*% matrix(beta, nrow=p))
  xb    <- as.vector(x %*% beta)
  alpha <- c(100e0, alpha, -100e0)
  # Must add 1 to y to compute P1 and P2 since index starts at 1, not 0
  P1    <- plogis(alpha[y + 1] + xb)
  P2    <- plogis(alpha[y + 2] + xb)
  Q     <- P1 - P2
  pq1   <- P1 * (1e0 - P1)
  pq2   <- P2 * (1e0 - P2)
  U     <- numeric(k + p)
  
  # Gradiant for intercepts
  for(m in 1 : k)
    U[m] <- sum((pq1 * (y == m) - pq2 * (y + 1 == m)) / Q)
  
  # Use element-wise multiplication then get the sum for each column
  # Element-wise = apply the same value of the weights to each row of x
  U[(k + 1) : (k + p)] <- colSums(x * (pq1 - pq2) / Q)

  U
}

Check that grad and grad2 yield the same answer and check their relative speeds.

set.seed(1)
n <- 50000; p <- 50; k <- 100
x <- matrix(rnorm(n * p), nrow=n)
y <- sample(0 : k, n, TRUE)
stopifnot(length(unique(y)) == k + 1)
alpha <- seq(-6, 6, length=k)
beta  <- runif(p, -0.5, 0.5)
tim(g1 = grad( alpha, beta, x, y),
    g2 = grad2(alpha, beta, x, y),
    reps = 5)   # creates Res

Per-run execution time in seconds, averaged over 5 runs 
   g1    g2 
0.918 0.072

m(Res$g1 - Res$g2)  # maximum absolute difference

[1] 4.183676e-11

Even though the streamlined code in grad2 required evaluating a few quantities that are known to be 0 or 1, its vectorization resulted in significantly faster R code. Later this will be compared with the speed of Fortran code.

Fortran Code for Gradient

Here is the central part of the Fortran code for computing the gradient vector. Fortran is blazing fast and easier to learn than C and C++, so more users may wish to translate some execution-time critical portions of their R code to Fortran 2018. R makes it easy to include Fortran code in packages, and it is also easy to include Fortran functions in RStudio sessions.

   u = 0_dp

    ! All obs with y=0
    ! The derivative of log expit(x) wrt x is expit(-x)
    ! Prob element is expit(-alpha(1) - lp)
    u(1) = - sum(wt(i0) * (1_dp - d(i0)))
    if(p > 0) then
      do l = 1, p
        u(k + l) = - sum(wt(i0) * x(i0, l) * (1_dp - d(i0)))
      end do
    end if
    ! All obs with y=k
    ! Prob element is expit(alpha(k) + lp)
    u(k) = u(k) + sum(wt(ik) * (1_dp - d(ik)))
    if(p > 0) then
      do l = 1, p
        u(k + l) = u(k + l) + sum(wt(ik) * x(ik, l) * (1_dp - d(ik)))
      end do
    end if
    ! All obs with 0 < y < k
    if(nb > 0) then
      do ii = 1, nb
        i = ib(ii)
        j = y(i)
        ! For p1, D() = 1 for alpha(j), 0 for alpha(j+1)
        ! For p2, D() = 0 for alpha(j), 1 for alpha(j+1)
        u(j)     = u(j)     + wt(i) * v1(i) / d(i)
        u(j + 1) = u(j + 1) - wt(i) * v2(i) / d(i)
        if(p > 0) then
          do l = 1, p
            u(k + l) = u(k + l) + wt(i) * x(i, l) * (v1(i) - v2(i)) / d(i)
          end do
        end if
      end do

This code can be streamlined using the extension approach:

    ealpha = [100d0, alpha, -100d0]
    p1 = expit(ealpha(y + 1) + lp)
    p2 = expit(ealpha(y + 2) + lp)
    q  = p1 - p2
    pq1 = p1 * (1_dp - p1)
    pq2 = p2 * (1_dp - p2)
     do j = 1, k
      u(j) = sum((pq1 * merge(1_dp, 0_dp, y     == j) - &
                  pq2 * merge(1_dp, 0_dp, y + 1 == j)) / (q / wt))
    end do
    if(p > 0) then
      do j = 1, p
        u(k + j) = sum(x(:, j) * (pq1 - pq2) / (q / wt))
      end do
    end if

But this code runs faster (this is the code tested below):

   do i = 1, n
      w = q(i) / wt(i)
      do j = 1, k
        if(y(i)     == j) u(j) = u(j) + pq1(i) / w
        if(y(i) + 1 == j) u(j) = u(j) - pq2(i) / w
      end do
      if(p > 0) then
        do j = 1, p
          u(k + j) = u(k + j) + x(i, j) * (pq1(i) - pq2(i)) / w
        end do
      end if
    end do

First let’s compare accuracy and speed of two ways of coding the gradient calculation in Fortran (click above to see both versions).

tim(Fortran  = fgrad (alpha, beta, x, y),
    Fortran2 = fgrad2(alpha, beta, x, y), reps=44/10 )

Per-run execution time in seconds, averaged over 4.4 runs 
    Fortran    Fortran2 
0.007500000 0.009545455

m(Res$Fortran - Res$Fortran2)

[1] 6.366463e-12

Though it produces the same result to within , the streamlined Fortran is slower than the longer Fortran code.

Run the R grad2 function defined above, and the Fortran routine included in the new rms package, for , predictors, and intercepts.

# Check agreement of R and Fortran code
tim(R       = grad2(alpha, beta, x, y),
    Fortran = fgrad(alpha, beta, x, y), reps=40 )

Per-run execution time in seconds, averaged over 40 runs 
       R  Fortran 
0.061750 0.007875

m(Res$R - Res$Fortran)

[1] 6.366463e-12

We see that the compiled Fortran code is faster than the R code. In a nutshell Fortran allows you to not worry about vectorizing calculations, allowing for simpler code (there are many functions in Fortran for vectorizing operations but these are used more for brevity than for speed).

A more vectorized version of the code, written by ChatGPT, gave completely incorrect results but only ran faster by a ratio of 0.84. After much prompting, ChatGPT could only get the right answer if it re-wrote the code to be very inefficient by using excessive loops.

ChatGPT’s Compact But Non-Working Code

As streamlined as this code is, it does not improve execution time over my R grad function, taking 1.1s to run on the data given above while yielding the wrong answer.

gradient_proportional_odds <- function(alpha, beta, X, Y) {
  n <- nrow(X)  # Number of observations
  p <- ncol(X)  # Number of predictors
  k <- length(alpha)  # Number of thresholds (max Y value)

  # Compute linear predictors
  eta <- X %*% beta  # n x 1 vector

  # Expand eta to match dimensions with alpha
  eta_matrix <- matrix(eta, n, k, byrow = FALSE)  # n x k matrix

  # Compute expit(alpha_y + eta) for all thresholds y
  eta_alpha <- eta_matrix + matrix(alpha, n, k, byrow = TRUE)
  expit_vals <- 1 / (1 + exp(-eta_alpha))  # n x k matrix of expit values

  # Compute probabilities for P(Y = y)
  expit_upper <- cbind(1, expit_vals)  # P(Y >= 0) = 1
  expit_lower <- cbind(expit_vals, 0)  # P(Y >= k+1) = 0
  prob_Y <- expit_upper[, 1:k] - expit_lower[, 1:k]  # P(Y = y)

  # Indicator matrix for observed Y
  Y_ind <- matrix(0, n, k)
  for (i in 1:k) Y_ind[, i] <- as.numeric(Y == (i - 1))

  # Compute weights (observed minus predicted probabilities)
  weights <- (Y_ind - prob_Y)

  # Gradients w.r.t. alpha
  grad_alpha <- colSums(weights)

  # Gradients w.r.t. beta
  d_expit <- expit_vals * (1 - expit_vals)  # Derivative of expit
  grad_beta <- numeric(p)
  for (j in 1:p) {
    grad_beta[j] <- sum(weights * d_expit * X[, j])
  }

  # Combine gradients
  grad <- c(grad_alpha, grad_beta)
  return(grad)
}

The hessian requires about a factor of more calculations than the gradient when computed inefficiently, so the Fortran code pays off even more when using hessian-based optimization algorithms or computing the final covariance matrix. The lrmll Fortran code called by the new lrm.fit, and ormll called by orm.fit, capitalize on the tri-band diagonal form of the hessian for the cumulative probability model. In rms 7.0-0 lrm.fit and orm.fit were completely re-written to benefit from Fortran code anx to use the Matrix package for more efficient sparse matrix handling.

Efficient Computation of the hessian for General Cumulative Probability Models

The lrmll Fortran subroutine computes the hessian very efficiently for the proportional odds model (logit link), making use of simplifications for the logistic model. Now consider general links. For is written in terms of the cumulative probability function by . We need all the second partial derivatives of the log of this difference in cumulative probabilities. Let’s simplify the expression to . ChatGPT provided the following results, which I simplified somewhat and corrected a sign error in .

Let
Let
Substitute for post differentiation
Divide all the second partial derivatives below by

For derivatives with respect only to , substitute for to get second partial derivatives with respect to and .

Let for and otherwise. For the interior values of , first consider which has probability . The second partial derivatives are, when multiplied by ,

Here stands for etc.

For the hessian elements are, when multiplied by ,

All these formulas are implemented in the ormll Fortran subroutine used in the 6.9-1 version of rms.

Use of Jacobians to Translate hessian on One Scale To Another Scale

Sometimes it’s easier to derive second derivatives on the original scale without logging, and using a Jacobian to translate to the other scale. ChatGPT provided the following Jacobian solution for our example. This is not used in the Fortran code because it might slow it down a bit. I believe that MASS::polr uses the Jacobian approach.

To systematically compute and represent the transformation of the second partial derivatives of into the second partial derivatives of , we organize the derivatives into a hessian matrix and apply the transformation rules using Jacobian operations.

Step 1: Define the hessian Matrix for

The hessian matrix of with respect to is:

Using the expressions for second partial derivatives derived earlier:

Thus:

Step 2: Define the Transformation for

We apply the chain rule:

This can be written compactly in matrix form.

Step 3: Matrix Transformation

Let:

be the hessian matrix of
be the scalar value of .

The hessian of is given by:

Explanation:

: Scales the hessian of by
: Accounts for the interaction of first derivatives of

Step 4: Components of

From earlier:

Thus:

Step 5: Compute

The outer product is:

Final Hessian of

The hessian of is:

Substitute and explicitly to get the full matrix. This provides the second derivatives of in terms of , its derivatives, and the structure of .

Check Speed of `NR`, `LM`, `nlminb`, and `glm.fit`

glm.fit is tailored to be efficient when Y is binary and there is no penalization, by using iteratively weighted least squares. How does it stack up against NR and nlminb? Let’s try fitting a large binary logistic model.

set.seed(1)
n <- 100000; p <- 100
X <- matrix(rnorm(n * p), nrow=n)
y <- sample(0 : 1, n, TRUE)
tim(NR      = lrm.fit(X, y, compstats=FALSE),
    LM      = lrm.fit(X, y, opt_method='LM',     compstats=FALSE),
    nlminb  = lrm.fit(X, y, opt_method='nlminb', compstats=FALSE),
    glm.fit = glm.fit(cbind(1, X), y), reps=3)

Per-run execution time in seconds, averaged over 3 runs 
      NR       LM   nlminb  glm.fit 
1.315667 1.196667 1.209333 1.639000

# transx=TRUE adds 2.3s to lrm.fit

Check Convergence Under Complete Separation

Consider a simple example where there is complete separation because the predictor values are identical to the response. In this case the MLE of the intercept is and the MLE of the slope is . The MLEs are approximated by , yielding predicted logits of and because the s are sufficiently close to 0.0 and 1.0.

plogis(-50, log=TRUE)

[1] -50

plogis( 50, log=TRUE)

[1] -1.92875e-22

exp(plogis(-50, log=TRUE))

[1] 1.92875e-22

exp(plogis( 50, log=TRUE))

[1] 1

See how 4 optimization algorithms fare with their default parameters and with some adjustments. In some of the trace output, the first floating point number listed is -2LL, and following that are the current estimates.

set.seed(1)
x <- sample(0 : 1, 20, TRUE)
y <- x
w <- try(lrm.fit(x, y, opt_method='NR', trace=1))  # default opt_method

Iteration:1  -2LL:26.92047  Max |gradient|:9.6  Max |change in parameters|:4.166667
Iteration:2  -2LL:4.704224  Max |gradient|:1.754066  Max |change in parameters|:2.249045
Iteration:3  -2LL:1.588994  Max |gradient|:0.616103  Max |change in parameters|:2.08089
Iteration:4  -2LL:0.5685801  Max |gradient|:0.2233137  Max |change in parameters|:2.028578
Iteration:5  -2LL:0.2071309  Max |gradient|:0.0817248  Max |change in parameters|:2.010364
Iteration:6  -2LL:0.07592908  Max |gradient|:0.03000809  Max |change in parameters|:2.003793
Iteration:7  -2LL:0.02789647  Max |gradient|:0.01103173  Max |change in parameters|:2.001393
Iteration:8  -2LL:0.01025764  Max |gradient|:0.004057316  Max |change in parameters|:2.000512
Iteration:9  -2LL:0.003772913  Max |gradient|:0.001492464  Max |change in parameters|:2.000188
Iteration:10  -2LL:0.001387888  Max |gradient|:0.000549028  Max |change in parameters|:2.000069
Iteration:11  -2LL:0.0005105632  Max |gradient|:0.0002019736  Max |change in parameters|:2.000025
Iteration:12  -2LL:0.0001878241  Max |gradient|:7.430157e-05  Max |change in parameters|:2.000009
Iteration:13  -2LL:6.90964e-05  Max |gradient|:2.733397e-05  Max |change in parameters|:2.000003
Iteration:14  -2LL:2.541911e-05  Max |gradient|:1.00556e-05  Max |change in parameters|:2.000001

w <- try(lrm.fit(x, y, opt_method='LM', trace=1))

Iteration:1  -2LL:4.726435  Max |gradient|:9.6  Max |change in parameters|:4.15697
Iteration:2  -2LL:1.596267  Max |gradient|:1.760569  Max |change in parameters|:2.249698
Iteration:3  -2LL:0.5711236  Max |gradient|:0.6183991  Max |change in parameters|:2.081206
Iteration:4  -2LL:0.2080488  Max |gradient|:0.2241376  Max |change in parameters|:2.028698
Iteration:5  -2LL:0.07626438  Max |gradient|:0.08202485  Max |change in parameters|:2.010408
Iteration:6  -2LL:0.0280195  Max |gradient|:0.03011806  Max |change in parameters|:2.003809
Iteration:7  -2LL:0.01030286  Max |gradient|:0.01107213  Max |change in parameters|:2.001399
Iteration:8  -2LL:0.003789542  Max |gradient|:0.004072171  Max |change in parameters|:2.000514
Iteration:9  -2LL:0.001394004  Max |gradient|:0.001497928  Max |change in parameters|:2.000189
Iteration:10  -2LL:0.0005128133  Max |gradient|:0.0005510379  Max |change in parameters|:2.00007
Iteration:11  -2LL:0.0001886518  Max |gradient|:0.0002027129  Max |change in parameters|:2.000026
Iteration:12  -2LL:6.94009e-05  Max |gradient|:7.457357e-05  Max |change in parameters|:2.000009
Iteration:13  -2LL:2.553113e-05  Max |gradient|:2.743404e-05  Max |change in parameters|:2.000003
Iteration:14  -2LL:9.392375e-06  Max |gradient|:1.009241e-05  Max |change in parameters|:2.000001

w <- try(lrm.fit(x, y, opt_method='nlminb',  trace=1))

  0:     26.920467: -0.405465  0.00000
  1:     5.9001181: -1.83776  3.67925
  2:     1.9469377: -2.99693  5.99700
  3:    0.69222129: -4.04687  8.09673
  4:    0.25163200: -5.06435  10.1316
  5:   0.092171572: -6.07067  12.1442
  6:   0.033854569: -7.07298  14.1489
  7:   0.012447190: -8.07383  16.1506
  8:  0.0045780905: -9.07414  18.1512
  9:  0.0016840535: -10.0743  20.1514
 10: 0.00061951084: -11.0743  22.1515
 11: 0.00022790289: -12.0743  24.1515
 12: 8.3840460e-05: -13.0743  26.1515
 13: 3.0843137e-05: -14.0743  28.1515
 14: 1.1346550e-05: -15.0743  30.1515
 15: 4.1741617e-06: -16.0743  32.1515
 16: 1.5355882e-06: -17.0743  34.1515
 17: 5.6491130e-07: -18.0743  36.1515
 18: 2.0781925e-07: -19.0743  38.1515
 19: 7.6452432e-08: -20.0743  40.1515
 20: 2.8125276e-08: -21.0743  42.1515
 21: 1.0346714e-08: -22.0743  44.1515
 22: 3.8063437e-09: -23.0743  46.1515
 23: 1.4002772e-09: -24.0743  48.1515
 24: 5.1513283e-10: -25.0743  50.1515
 25: 1.8950352e-10: -26.0743  52.1515
 26: 6.9714901e-11: -27.0743  54.1515
 27: 2.5648816e-11: -28.0743  56.1515
 28: 9.4360075e-12: -29.0743  58.1515
 29: 3.4692249e-12: -30.0743  60.1515
 30: 1.2789769e-12: -31.0743  62.1515
 31: 4.7073456e-13: -32.0743  64.1515
 32: 1.6875390e-13: -33.0743  66.1515
 33: 6.2172489e-14: -34.0743  68.1515
 34: 2.6645353e-14: -35.0743  70.1515
 35: 8.8817842e-15: -36.0743  72.1515
 36:     0.0000000: -37.0743  74.1515
 37:     0.0000000: -37.0743  74.1515

w <- try(lrm.fit(x, y, opt_method='glm.fit', trace=1))

Deviance = 3.368709 Iterations - 1
Deviance = 1.16701 Iterations - 2
Deviance = 0.4207147 Iterations - 3
Deviance = 0.1536571 Iterations - 4
Deviance = 0.0563787 Iterations - 5
Deviance = 0.02072057 Iterations - 6
Deviance = 0.007619969 Iterations - 7
Deviance = 0.002802865 Iterations - 8
Deviance = 0.001031067 Iterations - 9
Deviance = 0.0003793016 Iterations - 10
Deviance = 0.0001395364 Iterations - 11
Deviance = 5.133244e-05 Iterations - 12
Deviance = 1.888413e-05 Iterations - 13
Deviance = 6.947082e-06 Iterations - 14
Deviance = 2.555688e-06 Iterations - 15
Deviance = 9.401851e-07 Iterations - 16
Deviance = 3.458748e-07 Iterations - 17
Deviance = 1.272402e-07 Iterations - 18
Deviance = 4.680906e-08 Iterations - 19
Deviance = 1.722009e-08 Iterations - 20
Deviance = 6.33492e-09 Iterations - 21
Deviance = 2.330491e-09 Iterations - 22
Deviance = 8.573409e-10 Iterations - 23
Deviance = 3.15401e-10 Iterations - 24
Deviance = 1.160316e-10 Iterations - 25
Deviance = 4.268585e-11 Iterations - 26
Deviance = 1.570299e-11 Iterations - 27
Deviance = 5.782042e-12 Iterations - 28

w <- try(lrm.fit(x, y, opt_method='BFGS', trace=1))

initial  value 26.920467 
iter  10 value 0.000153
iter  20 value 0.000017
iter  30 value 0.000004
iter  40 value 0.000001
iter  50 value 0.000000
final  value 0.000000 
stopped after 50 iterations

nlminb took 37 iterations, going so far that the hessian matrix was singular. It should have stopped with 10.
glm.fit took 28 iterations; should have stopped with 10
BFGS: stopped after 50 iterations; should have stopped with 10

Now specify arguments to lrm.fit that are tuned to this task.

w <- lrm.fit(x, y, opt_method='nlminb',  abstol=1e-3, trace=1)

  0:     26.920467: -0.405465  0.00000
  1:     5.9001181: -1.83776  3.67925
  2:     1.9469377: -2.99693  5.99700
  3:    0.69222129: -4.04687  8.09673
  4:    0.25163200: -5.06435  10.1316
  5:   0.092171572: -6.07067  12.1442
  6:   0.033854569: -7.07298  14.1489
  7:   0.012447190: -8.07383  16.1506
  8:  0.0045780905: -9.07414  18.1512
  9:  0.0016840535: -10.0743  20.1514
 10: 0.00061951084: -11.0743  22.1515

w <- lrm.fit(x, y, opt_method='glm.fit', reltol=1e-3, trace=1)

Deviance = 3.368709 Iterations - 1
Deviance = 1.16701 Iterations - 2
Deviance = 0.4207147 Iterations - 3
Deviance = 0.1536571 Iterations - 4
Deviance = 0.0563787 Iterations - 5
Deviance = 0.02072057 Iterations - 6
Deviance = 0.007619969 Iterations - 7
Deviance = 0.002802865 Iterations - 8
Deviance = 0.001031067 Iterations - 9
Deviance = 0.0003793016 Iterations - 10
Deviance = 0.0001395364 Iterations - 11
Deviance = 5.133244e-05 Iterations - 12

w <- lrm.fit(x, y, opt_method='BFGS',    reltol=1e-4, trace=1)

initial  value 26.920467 
iter  10 value 0.000153
final  value 0.000100 
converged

The tolerance parameters are too large to use when infinite coefficients are not a problem.

Check Algorithms With k=1000

For timings that follow, compstats=FALSE is specified to lrm.fit so that we can focus on computationally efficiencies of various optimization algorithms. Some of the differences in run times may not seem to be consequential, but once extremely large datasets are analyzed or one needs to fit models in a bootstrap or Monte Carlo simulation loop, the differences in speed will matter.

When the number of distinct Y values is large, and this far exceeds the number of predictors (), the rms lrm and orm function are highly efficient. They take into account the sparsity of the intercept portion of the hessian matrix, which is tri-band diagonal and has only nonzero values for distinct Y values, taking the matrix’s symmetry into account. Outside of lrm, orm and SAS JMP, ordinal regression fitting software treats the hessian matrix as being and does not capitalize on sparsity.

Note that for Bayesian MCMC this is not an issue as posterior sampling does not need the hessian.

set.seed(1)
n <- 10000; k <- 1000
x <- rnorm(n); y <- sample(0:k, n, TRUE)
length(unique(y))

[1] 1001

tim(orm    = orm.fit(x, y, eps=1e-9,           compstats=FALSE),
    ormlm  = lrm.fit(x, y, opt_method='LM',     compstats=FALSE),
    bfgs   = lrm.fit(x, y, opt_method='BFGS',   compstats=FALSE, maxit=100),
    nlminb = lrm.fit(x, y, opt_method='nlminb', compstats=FALSE),
    nr     = lrm.fit(x, y, opt_method='NR',     compstats=FALSE),
    nlm    = lrm.fit(x, y, opt_method='nlm',    compstats=FALSE),
    polr   = MASS::polr(factor(y) ~ x, control=list(reltol=1e-10)),
    reps= 2)

Per-run execution time in seconds, averaged over 2 runs 
   orm  ormlm   bfgs nlminb     nr    nlm   polr 
0.0085 0.0140 0.0375 0.3215 0.0095 6.9605 6.4010

smod()

       deviance    max_abs_u iter
orm    137142.5 1.008812e-09    2
ormlm  137142.5 5.287811e-04    3
bfgs   137142.5 2.376188e+00    4
nlminb 137142.5 2.046306e-09    3
nr     137142.5 7.625981e-05    2
nlm    137142.5 7.625981e-05    1
polr   137142.5           NA   NA

Maximum |difference in coefficients|, Maximum |relative difference|
 worst ratio of covariance matrices

         Comparison Max |difference| Max |rel diff| Cov ratio
1     orm vs. ormlm     4.528191e-06   2.116413e-05  1.020062
2      orm vs. bfgs     6.991343e-06   3.950698e-04          
3    orm vs. nlminb     2.194293e-15   8.487308e-15  1.000000
4        orm vs. nr     1.376879e-15   5.475644e-15  1.000000
5       orm vs. nlm     4.136309e-07   3.812140e-07  1.001368
6      orm vs. polr     9.637588e-06   1.359060e-05          
7    ormlm vs. bfgs     3.000666e-06   3.747458e-04          
8  ormlm vs. nlminb     4.528191e-06   2.116413e-05  1.020062
9      ormlm vs. nr     4.528191e-06   2.116413e-05  1.020062
10    ormlm vs. nlm     4.219032e-06   2.103439e-05  1.021457
11   ormlm vs. polr     8.539945e-06   3.049453e-05          
12  bfgs vs. nlminb     6.991343e-06   3.950698e-04          
13      bfgs vs. nr     6.991343e-06   3.950698e-04          
14     bfgs vs. nlm     6.776204e-06   3.949523e-04          
15    bfgs vs. polr     9.425718e-06   4.030520e-04          
16    nlminb vs. nr     8.695423e-16   3.063504e-15  1.000000
17   nlminb vs. nlm     4.136309e-07   3.812140e-07  1.001368
18  nlminb vs. polr     9.637588e-06   1.359060e-05          
19       nr vs. nlm     4.136309e-07   3.812140e-07  1.001368
20      nr vs. polr     9.637588e-06   1.359060e-05          
21     nlm vs. polr     9.231301e-06   1.326760e-05

nlminb is slower then NR because to run nlminb requires converting the Hession from a sparse Matrix into a regular dense matrix.

Check Timing and Agreement for n=100000, k=10, p=5

Check timing and calculation agreement for n=100000, 10 intercepts, 5 predictors.

set.seed(1)
n <- 100000
y <- sample(0 : 10, n, TRUE)
x1 <- rnorm(n)
x2 <- rnorm(n)
x3 <- rnorm(n)
x4 <- rnorm(n)
x5 <- rnorm(n)
X <- cbind(x1, x2, x3, x4, x5)

tim(old.lrm.fit = olrm(X, y, eps=1e-7),
    nr          = lrm.fit(X, y, opt_method='NR', compstats=FALSE),
    lm          = orm.fit(X, y, opt_method='LM', compstats=FALSE),
    nlm         = lrm.fit(X, y, opt_method='nlm', compstats=FALSE),
    nlminb      = lrm.fit(X, y, opt_method='nlminb', compstats=FALSE, transx=TRUE),
    nlminb.notransx = lrm.fit(X, y, opt_method='nlminb', compstats=FALSE, transx=FALSE),
    bfgs        = lrm.fit(X, y, opt_method='BFGS', compstats=FALSE),
    bfgs.reltol = lrm.fit(X, y, opt_method='BFGS',
                          compstats=FALSE, reltol=1e-12),
    polr        = MASS::polr(factor(y) ~ X,
                             control=list(reltol=1e-10)),
    reps = 5)

Per-run execution time in seconds, averaged over 5 runs 
    old.lrm.fit              nr              lm             nlm          nlminb 
         0.1468          0.0838          0.0732          0.4820          0.1082 
nlminb.notransx            bfgs     bfgs.reltol            polr 
         0.0774          0.3912          0.5786          2.4018

smod()

                deviance    max_abs_u iter
old.lrm.fit     479562.9 6.575185e-12   NA
nr              479562.9 1.992454e-07    3
lm              479562.9 4.246613e-05    3
nlm             479562.9 5.051226e-02    1
nlminb          479562.9 1.992456e-07    3
nlminb.notransx 479562.9 1.992452e-07    3
bfgs            479562.9 6.987317e-01    7
bfgs.reltol     479562.9 3.683980e-05   19
polr            479562.9           NA   NA

Maximum |difference in coefficients|, Maximum |relative difference|
 worst ratio of covariance matrices

                        Comparison Max |difference| Max |rel diff| Cov ratio
1               old.lrm.fit vs. nr     5.404639e-16   9.835782e-15  1.000000
2               old.lrm.fit vs. lm     1.964277e-09   3.686249e-08  1.000000
3              old.lrm.fit vs. nlm     3.004074e-06   2.122140e-05  1.000202
4           old.lrm.fit vs. nlminb     1.053798e-11   4.463472e-11  1.000000
5  old.lrm.fit vs. nlminb.notransx     1.053800e-11   4.461566e-11  1.000000
6             old.lrm.fit vs. bfgs     9.057305e-06   7.174198e-04          
7      old.lrm.fit vs. bfgs.reltol     5.712939e-08   1.703831e-06          
8             old.lrm.fit vs. polr     5.977469e-06   2.409730e-03          
9                        nr vs. lm     1.964277e-09   3.686250e-08  1.000000
10                      nr vs. nlm     3.004074e-06   2.122140e-05  1.000202
11                   nr vs. nlminb     1.053798e-11   4.462649e-11  1.000000
12          nr vs. nlminb.notransx     1.053799e-11   4.460743e-11  1.000000
13                     nr vs. bfgs     9.057305e-06   7.174198e-04          
14              nr vs. bfgs.reltol     5.712939e-08   1.703831e-06          
15                     nr vs. polr     5.977469e-06   2.409730e-03          
16                      lm vs. nlm     3.002323e-06   2.123986e-05  1.000202
17                   lm vs. nlminb     1.954251e-09   3.688524e-08  1.000000
18          lm vs. nlminb.notransx     1.954251e-09   3.688522e-08  1.000000
19                     lm vs. bfgs     9.056084e-06   7.173837e-04          
20              lm vs. bfgs.reltol     5.903326e-08   1.726185e-06          
21                     lm vs. polr     5.978924e-06   2.409739e-03          
22                  nlm vs. nlminb     3.004063e-06   2.122136e-05  1.000202
23         nlm vs. nlminb.notransx     3.004063e-06   2.122136e-05  1.000202
24                    nlm vs. bfgs     8.867603e-06   7.344671e-04          
25             nlm vs. bfgs.reltol     3.059332e-06   2.243214e-05          
26                    nlm vs. polr     7.995997e-06   2.398944e-03          
27      nlminb vs. nlminb.notransx     3.778083e-17   2.518821e-14  1.000000
28                 nlminb vs. bfgs     9.057301e-06   7.174198e-04          
29          nlminb vs. bfgs.reltol     5.713988e-08   1.703864e-06          
30                 nlminb vs. polr     5.977476e-06   2.409730e-03          
31        nlminb.notransx vs. bfgs     9.057301e-06   7.174198e-04          
32 nlminb.notransx vs. bfgs.reltol     5.713988e-08   1.703864e-06          
33        nlminb.notransx vs. polr     5.977476e-06   2.409730e-03          
34            bfgs vs. bfgs.reltol     9.069540e-06   7.167658e-04          
35                   bfgs vs. polr     1.347117e-05   2.800575e-03          
36            bfgs.reltol vs. polr     5.951212e-06   2.410693e-03

Other Speed Tests

n=1,000,000, p=100 predictors, k=50 intercepts

lrm.fit: 13.5s (13.6 with opt_method=nlminb, 120s with opt_method='BFGS')
orm.fit: 9s
MASS::polr: 70s without the hessian

n=300,000, p=20, k=299,999

lrm.fit: 2.25s
orm.fit: 2s

Execution time is proportional to

Check Impact of `initglm` and `transx`

Generate a sample of 500 with 30 predictors and 269 levels of Y where a subset of the predictors are strongly related to Y and there are collinearities.

set.seed(1)
n <- 500
p <- 30
x <- matrix(runif(n * p), nrow=n)
x[, 1:8] <- x[, 1:8] + 2 * x[, 9]
s <- varclus(~ x)$hclust
plot(as.dendrogram(s), horiz=TRUE, axes=FALSE,
     xlab=expression(paste('Spearman ', rho^2)))
rh <- seq(0, 1, by=0.1)
axis(1, at=1 - rh, labels=format(rh))

y <- x[, 1] + 2 * x[, 2] + 3 * x[, 3] + 4 * x[, 4]+ 5 * x[, 5] +
     3 * runif(n, -1, 1)
y <- round(y, 1)
length(unique(y))

[1] 269

f <- function(..., opt_method='NR')
  lrm.fit(x, y, compstats=FALSE, opt_method=opt_method, maxit=1000, ...)
# f(initglm=TRUE) (using nlminb) would not work: NA/NaN gradient evaluation
# This did not happen without collinearities
tim(default          = f(),
    transx           = f(transx=TRUE),
    nlm              = f(opt_method='nlm'),
    bfgs             = f(opt_method='BFGS'),
    nlminb           = f(opt_method='NR'),
    reps = 10 )

Per-run execution time in seconds, averaged over 10 runs 
default  transx     nlm    bfgs  nlminb 
 0.0136  0.0163  0.5946  0.2671  0.0210

smod()

        deviance    max_abs_u iter
default 3879.848 6.112532e-05    7
transx  3879.848 6.112532e-05    7
nlm     3879.848 6.112532e-05    6
bfgs    3879.848 5.367424e-03  510
nlminb  3879.848 6.112532e-05    7

Maximum |difference in coefficients|, Maximum |relative difference|
 worst ratio of covariance matrices

           Comparison Max |difference| Max |rel diff| Cov ratio
1  default vs. transx     7.418667e-15   8.758898e-16  1.000000
2     default vs. nlm     1.063035e-05   5.990158e-07  1.000291
3    default vs. bfgs     1.902524e-05   4.646345e-06          
4  default vs. nlminb     0.000000e+00   0.000000e+00  1.000000
5      transx vs. nlm     1.063035e-05   5.990158e-07  1.000291
6     transx vs. bfgs     1.902524e-05   4.646345e-06          
7   transx vs. nlminb     7.418667e-15   8.758898e-16  1.000000
8        nlm vs. bfgs     1.629260e-05   4.555452e-06          
9      nlm vs. nlminb     1.063035e-05   5.990158e-07  1.000291
10    bfgs vs. nlminb     1.902524e-05   4.646345e-06

`lrm.fit` vs. `orm.fit` as k

The fitting function for rms::orm, orm.fit, uses sparse hessian matrices so that the computation time is roughly proportional to where is the number of intercepts and is the number of predictors. Computation of the hessian in lrm.fit needs about computations, but some parts of the computation are faster and there is some overhead of handling sparse matrices in orm.fit. Let’s explore execution time as a function of when and varies. There should be very little difference.

if(! file.exists('breakeven.rds')) {
  set.seed(1)
  n  <- 10000
  p  <- 30
  ks <- seq(100, 10000, by=200)
  l  <- length(ks)
  t1 <- t2 <- d <- numeric(l)
  x  <- matrix(rnorm(n * p), nrow=n)
  for(i in 1 : l) {
    cat(ks[i], ' ')
    y     <- rep(0 : ks[i], length=n)
    t1[i] <- stim(for(j in 1:20) f <- rms::lrm.fit(x, y)) / 20
    t2[i] <- stim(for(j in 1:20) g <- rms::orm.fit(x, y)) / 20
    d [i] <- m(coef(f) - coef(g))
  }
  w <- llist(ks, t1, t2, d)
  saveRDS(w, 'breakeven.rds')
} else {
  w <- readRDS('breakeven.rds')
  ks <- w$ks; t1 <- w$t1; t2 <- w$t2; d <- w$d
}
# Make sure coefficients agree
range(d)

[1] 8.881784e-16 1.532996e-12

plot(ks, t1, type='b', xlab='k', ylab='Time, seconds', ylim=c(0.02, 0.18))
points(ks, t2, col='red')
lines (ks, t2, col='red')

Execution time for both functions is linear in k. orm is consistently a little faster than lrm. Since the code in the orm.fit Fortran ormll subroutine is more general, currently implementing 5 link functions, there is no real reason to maintain separate code. In the future I plan to merge the functions to minimize duplication, and having an lrm front-end for orm for backward compatibility.

Better Understanding Convergence with BFGS Optimizer

Using the same simulated data just used with k=20, use BFGS to fit an ordinal model with relative tolerance varying from to . Estimates are compared to orm. In addition to comparing parameter estimates as done above we also compute differences in units of standard errors as computed by orm.

set.seed(1)
y  <- sample(0:20, n, TRUE)
g  <- orm(y ~ x, eps=1e-10)
se <- sqrt(diag(vcov(g, intercepts='all')))
length(se)

[1] 50

if(file.exists('bfgs-reltol.rds')) d <- readRDS('bfgs-reltol.rds') else {
  d <- NULL
  for(i in 2 : 20) {
    cat(i, '')
    s <- stim({
      for(j in 1:5)
        f <- lrm.fit(x, y, compstats=FALSE,
                     opt_method='BFGS',
                     maxit=1000,
                     reltol=10^(-i))
    } )
  
  w <- data.frame(i, elapsed=s / 5, maxu=m(f$u),
                  maxbeta=m(coef(g) - coef(f)),
                  maxbeta.per.se=m((coef(g) - coef(f)) / se), 
                  deviance=tail(f$deviance, 1),
                  iter=tail(f$iter, 1))
  d <- rbind(d, w)
}
rownames(d) <- NULL
saveRDS(d, 'bfgs-reltol.rds')
}
d

    i elapsed         maxu      maxbeta maxbeta.per.se deviance iter
1   2  0.0150 1.474752e+02 2.559219e-02   1.485374e+00 60853.10    1
2   3  0.0156 1.474752e+02 2.559219e-02   1.485374e+00 60853.10    1
3   4  0.0520 4.754120e+01 8.310188e-03   4.823241e-01 60843.74    3
4   5  0.0676 2.731752e+01 6.488538e-03   3.765954e-01 60843.47    4
5   6  0.1164 2.570406e+01 4.628291e-03   1.152693e-01 60842.69   10
6   7  0.1634 3.232073e+00 4.015687e-03   9.284606e-02 60842.60   16
7   8  0.4220 1.849727e+00 4.627223e-04   1.288221e-02 60842.57   55
8   9  0.4202 4.694068e-01 8.486176e-04   2.144243e-02 60842.57   53
9  10  0.5654 1.027168e-01 1.576543e-04   3.340893e-03 60842.57   75
10 11  0.3670 3.118309e-02 5.997566e-05   1.292625e-03 60842.57   52
11 12  0.5104 1.435433e-02 6.238079e-06   1.344461e-04 60842.57   71
12 13  0.3986 9.525472e-03 1.005773e-05   2.167691e-04 60842.57   56
13 14  0.3882 1.054183e-03 1.688144e-06   3.638369e-05 60842.57   57
14 15  0.5556 1.669547e-03 1.222338e-06   2.832486e-05 60842.57   70
15 16  0.4710 4.720818e-04 1.373869e-07   7.956884e-06 60842.57   60
16 17  0.3992 4.720818e-04 1.373869e-07   7.956884e-06 60842.57   60
17 18  0.3950 4.720818e-04 1.373869e-07   7.956884e-06 60842.57   60
18 19  0.4126 4.720818e-04 1.373869e-07   7.956884e-06 60842.57   60
19 20  0.4522 4.720818e-04 1.373869e-07   7.956884e-06 60842.57   60

z <- c(deviance=8, beta=10, beta.se=6, grad=15)  # minimum i for which success achieved
h <- function() abline(v=z, col=gray(0.60))
h <- function() {}    # remove this line to show reference lines
par(mfrow=c(2, 3), mar=c(4, 4, 1, 1), las=1, mgp=c(2.8, .45, 0))
with(d, {
  plot(i, elapsed, type='l'); h()
  plot(i, maxu, type='l', log='y'); h()
  plot(i, maxbeta, type='l', log='y'); h()
  plot(i, maxbeta.per.se, type='l'); h()
  plot(i, deviance - 3879, type='l', log='y'); h()
  plot(i, iter, type='l', log='y'); h()
})

NULL

See that stochastic convergence, as judged by deviance, occurs by the time the relative tolerance is , by to control maximum absolute parameter difference, by to get parameter estimates to within 0.06 standard error, and by to achieve gradients in absolute value.

When using BFGS a recommendation is to use a relative tolerance of to nail down estimates to the extent that it matters precision-wise, and use to achieve reproducibility.

Recall that BFGS is only appealing when the number of intercepts is large, you don’t need the covariance matrix, and you are not using orm.

Matrix Inversion

The information matrix (negative hessian) must be inverted to compute the variance-covariance matrix. The default inversion method in R is the solve function, which defaults to using the LU decomposition. This is a fast algorithm, but the Cholesky decomposition is faster and behaves as well as LU numerically. Another approach uses the QR decomposition, implemented in the qr.solve function. Let’s compare speed and accuracy of the three approaches when applied to a high-dimensional almost-singular matrix.

# ChatGPT created this function to generate an almost-singular symmetric
# positive definite matrix of dimension p x p
genas <- function(p, epsilon = 1e-8) {
  # Generate a random symmetric positive definite matrix
  A <- matrix(rnorm(p^2), p, p)
  A <- A + t(A) + p * diag(p)  # Symmetric and positive definite
  
  # Modify eigenvalues to make the matrix almost singular
  eigen_decomp <- eigen(A)
  eigen_decomp$values[p] <- epsilon  # Set the smallest eigenvalue close to zero
  eigen_decomp$vectors %*% diag(eigen_decomp$values) %*% t(eigen_decomp$vectors)
}
set.seed(3)
x <- genas(750, 1e-9)
x[1:3, 1:3]

           [,1]        [,2]        [,3]
[1,] 748.051108  -1.0242634   0.3929600
[2,]  -1.024263 747.2599805   0.9261079
[3,]   0.392960   0.9261079 745.6809913

# LU method
stime(for(i in 1:3) a <- solve(x))

Elapsed time: 0.489s

mad(x, solve(a))       # reverse the inversion and compare to x

         mad       relmad 
0.0001918367 0.0006802723

m(diag(750) - x %*% a)  # compare x inverse * x to identity matrix

[1] 2.636021e-05

# QR
stime(for(i in 1:3) b <- qr.solve(x, tol=1e-13))

Elapsed time: 1.576s

mad(x, qr.solve(b, tol=1e-13))

         mad       relmad 
0.0002055301 0.0007529110

m(diag(750) - x %*% b)

[1] 3.14796e-05

# Cholesky
stime(for(i in 1:3) ch <- chol2inv(chol(x)))

Elapsed time: 0.331s

mad(x, chol2inv(chol(ch)))  # mean |difference| and mean relative difference

         mad       relmad 
0.0002321373 0.0008178099

m(diag(750) - x %*% ch)   # mean |difference|

[1] 2.07157e-05

QR takes significantly longer and offers no accuracy advantange. Inversion via Cholesky decomposition was almost twice as fast as LU, though both methods took less than of a second to invert a matrix. Cholesky was a little more accurate in getting the product of the original matrix and its inverse closer to an identity matrix. Cholesky was very slightly worse in recovering the original matrix by inverting its inverse.

One of the reasons lrm.fit and orm.fit are more efficient in rms 7.0 is that the entire information matrix is not inverted upon convergence when creating the final fit object. Much use is made of the Matrix package for efficient storage and computation of sparse matrices, and only the 3 minimal submatrices that make up the information matrix are stored. These are operated on quite generally by the new rms infoMxop function, which is called by the vcov method to invert pieces of the information matrix only as needed. During Newton-type updating, the Matrix solve function is used, which is quite fast as it uses sparse representations and does not actually invert the Hession but solves for the inverse of the hessian multiplied by the gradient vector.

Here is an example where only elements 10:20 from the inverse of a 1000 x 1000 matrix are obtained. This type of coding is used in infoMxop.

# Create a 1000 x 1000 symmetric positive definite matrix
set.seed(1)
x <- matrix(rnorm(10000 * 1000), ncol=1000)
v <- crossprod(x)
i <- 10:20    # submatrix of v inverse that we want
l <- length(i)
w <- matrix(0, 1000, l)
w[cbind(i, 1:l)] <- 1    # w is all zeros except for 1 in i elements
sum(w)

[1] 11

stime(vi <- solve(v, w)[i, , drop=FALSE])

Elapsed time: 0.118s

dim(vi)

[1] 11 11

stime(vi_slow <- solve(v))

Elapsed time: 0.367s

range(vi - vi_slow[i, i])

[1] 0 0

What about inverting the kind of sparse matrices that ordinal models deal with? Let’s build one using the Matrix package and rms::infoMxop.

require(Matrix)
p <- 200
k <- 20000
set.seed(1)
w <- list(a = cbind(runif(k), runif(k)),
          b = v[1:p, 1:p],
          ab= matrix(runif(k * p), k, p) )
stime(z <- infoMxop(w))

Elapsed time: 0.15s

object.size(w)

32641088 bytes

object.size(z)

97282280 bytes

# Sparse representation of intercept components used by infoMxop
ia <- Matrix::bandSparse(k, k=c(0,1), diagonals=w$a, symmetric=TRUE)
object.size(ia)

561720 bytes

# Time to invert this sparse matrix
stime(via <- Matrix::solve(ia))

Elapsed time: 1.984s

dim(via)

[1] 20000 20000

# The inverse of a tri-band diagonal matrix is dense but can be represented efficiently
object.size(via)

383980824 bytes

length(via@x)

[1] 31991591

# Compute size needed if did not make use of sparsity
8 * (p + k) ^ 2

[1] 3264320000

# Get covariate portion of inverted matrix
stime(ub <- infoMxop(w, i='x'))

Elapsed time: 0.371s

# Get the first intercept and beta portion of the inverse
i <- c(1, (k + 1) : (k + p))
stime(u  <- infoMxop(w, i=i))    # 26s for i=one element

Elapsed time: 48.088s

dim(u); dim(ub); range(u[-1, -1] - ub)

[1] 201 201

[1] 200 200

[1] -5.009692e-17  4.220257e-17

# Don't try this:  infoMxop(w, invert=TRUE))

The specialized method with i='x' for getting just the portion of the inverse corresponding to the s is very fast. Otherwise there are speed challenges but the sparse representation does allow the overall inverse to be computed, something not possible with naive matrix calculations.

Now consider execution time for computing the standard errors of predicted values . Consider a predicted value of the form involving a single intercept . Let be the variance-covariance matrix for where is a vector of length . Let the matrix consisting of a column of ones followed by columns of predictor settings. is the number of observations for which predictions are sought. Standard errors of interest are square roots of the diagonal of the variance-covariance matrix for , where is an matrix.

This also applies to contrasts where differences in are substituted for .

rms::infoMxop makes computations such as efficient using the following strategy. Let be a matrix that is mainly zeros but with ones in positions that indicate which elements of to compute. Let denote the information matrix for the entire model, with rows and columns. Concentrate on computation of . Instead of computing by itself, compute only the needed elements of it by computing solve(I, J) to get . But we quickly want to post-multiply by so use solve(I, JX'). Let’s see how fast this is when there are 299,999 intercepts.

set.seed(1)
n <- 300000; p <- 10; x <- matrix(runif(n * p), ncol=p); y <- 1 : n
k <- length(unique(y)) - 1
f <- orm(y ~ x)
# Get intercept number corresponding to median of y
j <- f$interceptRef
# Compute all parameter numbers needed
h    <- c(j, (k + 1) : (k + p))
info <- f$info.matrix
mdim(info)   # show dimensions of submatrices

             a  b     ab
rows    299999 10 299999
columns      2 10     10

# Form covariate values for 2 observations for predicting
X <- rbind(c(1, rep(.2, p)), c(1, rep(.6, p)))
X

     [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10] [,11]
[1,]    1  0.2  0.2  0.2  0.2  0.2  0.2  0.2  0.2   0.2   0.2
[2,]    1  0.6  0.6  0.6  0.6  0.6  0.6  0.6  0.6   0.6   0.6

# Get VX' the slow way
# system.time(infoMxop(info, invert=TRUE)[h, h] %*% t(X))   5.4s for k=9999
et <- stime(a <- infoMxop(info, i=h, B=t(X)))

Elapsed time: 0.315s

                   [,1]          [,2]
y>=150000  1.933352e-04 -4.675176e-05
x[1]      -3.589966e-05  1.195571e-05
x[2]      -3.618740e-05  1.220834e-05
x[3]      -3.635284e-05  1.205578e-05
x[4]      -3.595976e-05  1.195893e-05
x[5]      -3.587225e-05  1.212976e-05
x[6]      -3.568398e-05  1.180846e-05
x[7]      -3.607189e-05  1.206577e-05
x[8]      -3.581018e-05  1.194599e-05
x[9]      -3.613558e-05  1.194314e-05
x[10]     -3.610595e-05  1.212468e-05

stime(infoMxop(info, i=h))     # time required to compute needed submatrix of V

Elapsed time: 0.42s

stime(infoMxop(info, i=100))   # time required to retrieve a single intercept

Elapsed time: 0.308s

stime(infoMxop(info, i='x'))   # time to get beta part of v

Elapsed time: 0.129s

0.31 seconds to get the variance of predicted values when there are 299999 intercepts is quite sufficient! The time required to compute the portion of the inverse needed is only 0.1s longer however.

Note that in the source code for lrm and orm you’ll see a shortcut for computing the diagonal elements:

nx  <- ncol(X)
X   <- cbind(1, X)
nrp <- num.intercepts(f)
v   <- infoMxop(info, i=c(f$interceptRef, (nrp + 1) : (nrp + nx)), B=t(X))
se  <- drop(sqrt((t(v) * X) %*% rep(1, nx + 1)))

What is Fast and What is Slow When is Large

For continuous Y when there is a large number of intercepts, here is a breakdown of what kind of computations involving ordinal regression models are fast:

solving for the MLEs
computing the covariance matrix for the s alone and using computing it for and a small number of intercepts
any assessment that is relative (e.g., odds ratios as opposed to absolute risk estimates)
- contrasts
- Wald tests
- likelihood ratio tests (which don’t require covariance matrices)
- standard errors (SEs) and confidence bands for differences on the link (e.g., logit) scale
- predicted absolute quantities (exceedance probabilities, cell probabilities, means, quantiles) without SEs or confidence intervals

Slow operations for very large , e.g., :

computing the covariance matrix for all the intercepts or for all parameters combined

For some computations it will be faster to bootstrap the model fit rather than to compute SEs and CLs. The rms bootcov function does this efficiently since it uses lrm.fit or orm.fit with compstats=FALSE to streamline the computation of MLEs from bootstrap samples.

bootcov needs all the intercepts to be represented in all bootstrap samples. To minimally group Y values to make this happen, see the new rms ordGroupBoot function.

But for some “absolute” computations, run time is still exceptionally fast for large , because the entire information matrix does not need inverting, but instead the inverse is multiplied by a vector as was done above, so that solve can be used to quickly solve a system of equations instead of fully inverting. As an example let’s time the calculation of the estimation of mean Y without confidence limits, and then with them, for . The Mean function uses the -method to estimate the needed standard error for the normal approximation for the confidence interval for a covariate-specific population mean..

set.seed(1)
n <- 15000
y <- 1:n
x1 <- rnorm(n)
x2 <- rnorm(n)
x3 <- rnorm(n)
dd <- datadist(x1, x2, x3); options(datadist='dd')
f <- orm(y ~ x1 + x2 + x3, x=TRUE)
d <- data.frame(x1=0, x2=0, x3=0)
X <- predict(f, d, type='x')  # need original design matrix for accurate CLs of means
X

  x1 x2 x3
1  0  0  0

M <- Mean(f)
stime(print(M(predict(f, d))))

       1 
7500.703 
Elapsed time: 0.022s

stime(print(M(predict(f, d), conf.int=0.95, X=X)))

       1 
7500.703 
attr(,"limits")
attr(,"limits")$lower
     1 
7431.4 

attr(,"limits")$upper
       1 
7570.006 

Elapsed time: 0.045s

stime(print(Predict(f, x1=0, x2=0, x3=0, fun=M)))

  x1 x2 x3     yhat  lower    upper
1  0  0  0 7500.703 7431.4 7570.006

Response variable (y):  

Limits are 0.95 confidence limits
Elapsed time: 0.048s

This is fast because the limiting step is like this, inside the M R function:

info <- f$info.matrix
mdim(info)   # show dimensions of 3 submatrices

            a b    ab
rows    14999 3 14999
columns     2 3     3

np   <- sum(dim(info$ab))
np  # total no. parameters = # rows and cols of info matrix

[1] 15002

# Multiply info inverse times B
stime(infoMxop(info, B=matrix(runif(np), ncol=1)))

Elapsed time: 0.007s

Other Resources

Video by Richard McElreath explaining numerical accuracy issues in log likelihood calculations.
R CRAN Task Views on optimizers
Calculating OLS regression coefficients by TB Volker, which contains excellent background information about matrix algebra and timing and memory usage of various algorithms in R
Vignettes by John Nash et al.
R ucminf package
R nloptr package
R code: lrm and lrm.fit
Fortran code: lrmll
A cornucopia of maximum likelihood algorithms by K Lange, XJ Li, Hua Zhou, 2025.

Computing Environment

grateful::cite_packages(pkgs='Session', output='paragraph', out.dir='.',
    cite.tidyverse=FALSE, omit=c('grateful', 'ggplot2'))

We used R version 4.5.1 (R Core Team 2025) and the following R packages: Hmisc v. 5.2.4 (Harrell Jr 2025a), Matrix v. 1.7.3 (Bates, Maechler, and Jagan 2025), orms v. 1.0.0 (Harrell Jr 2010), rms v. 8.1.0 (Harrell Jr 2025b), survival v. 3.8.3 (Terry M. Therneau and Patricia M. Grambsch 2000; Therneau 2024).

The code was run on macOS Sequoia 15.6 on a Macbook Pro M2 Max, running on a single core.

References

Bates, Douglas, Martin Maechler, and Mikael Jagan. 2025. Matrix: Sparse and Dense Matrix Classes and Methods. https://doi.org/10.32614/CRAN.package.Matrix.

Harrell Jr, Frank E. 2010. orms: Regression Modeling Strategies. http://biostat.mc.vanderbilt.edu/rms.

———. 2025a. Hmisc: Harrell Miscellaneous. https://hbiostat.org/R/Hmisc/.

———. 2025b. rms: Regression Modeling Strategies. https://hbiostat.org/R/rms/.

R Core Team. 2025. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. https://www.R-project.org/.

Terry M. Therneau, and Patricia M. Grambsch. 2000. Modeling Survival Data: Extending the Cox Model. New York: Springer.

Therneau, Terry M. 2024. A Package for Survival Analysis in r. https://CRAN.R-project.org/package=survival.

Reuse

CC BY 4.0

Ordinal State Transition Models as a Unifying Risk Prediction Framework

Frank Harrell — Mon, 18 Nov 2024 06:00:00 GMT

Event:
- International Chinese Statistical Association Applied Statistics Symposium, Nashville, Tennessee USA 2024-06-17
- CANSSI Ontario STatistics Seminars (CAST), Virtual, 2024-11-18
Slides

Adjudication and Statistical Efficiency

Frank Harrell — Thu, 17 Oct 2024 05:00:00 GMT

Background

In clinical and epidemiologic studies one is frequently tasked with maximizing accuracy when assessing the presence of clinical conditions (symptoms, diagnoses, syndromes, etc.) or verifying outcome events such as stroke, myocardial infarction, or death from a specific cause. Prospective studies have the advantage of standardizing definitions of clinical conditions, minimizing bias, and being honest about disagreements about clinical designations. Many studies have clinical endpoint committees or adjudication committees. Statistical efficiency and completeness of reporting are optimized by having as many committee members as feasible, and having the members operate as independently as possible.

Statistical efficiency also comes from minimizing forced choices and utilizing gray zones. For example, if a study has only one adjudicator, and this clinical expert is uncertain about some of the designations, it is best for her to code determinations using at least one level of gray. The way to understand why this is more statistically efficient than having forced choices is to consider a 3-level (negative, uncertain, positive) clinical outcome that is being correlated with a 5-level severity of a symptom. Uncertain outcomes may occur more often for patients having a middle symptom severity. Making use of 3 levels of outcome will capitalize on this to increase power.

Sometimes the clinical condition needs to be used not as a multilevel ordinal outcome but is instead used in subsetting patients. For example, one may want to analyze a subset of the cohort consisting of patients designated as having a certain clinical syndrome at baseline. It is not hard to analyze subsets when the subsetting is uncertain. For example, if one translated an adjudication to the probability the patient has a syndrome, one can easily use multiple imputation to analyze subsets under uncertainties. If a given patient has a probability of 0.6 of having syndrome X, 10 imputations of the binary syndrome can be generated. In the long run, of the imputations will be positive for the syndrome. The needed subset analysis can be done by including, for each of the multiple imputations, all the patients imputed to be positive for X. By repeating this process over, say, 10 multiple imputations, noise in this process will average out and one-time forced choice classification is unnecessary.

Even if one does not want to use multiple imputation or Bayesian models to account for adjudication uncertainties, it is important to design the adjudications to lead to an optimum final negative/positive designation.

A Hierarchy of Statistical Information and Power

Besides having independent adjudicators, statistical information is maximized when one delays forced-choice designations as much as possible and respects gray zones to the extent possible. Here is a hierarchy of statistical information/efficiency/power from highest to lowest, for various strategies.

Have each adjudicator record the probability the patient is in the clinical category of interest, then average these probabilities to yield a final result that is used in analyses. When the clinical category is used as an outcome variable, ordinal regression may be used in the final analysis. This can be used to estimate the probability that the outcome is at a certain level or higher, for any and for any level of baseline variables.
Classify the patient as negative/positive depending on whether this average probability the condition exists exceeds a pre-specified level.
Have each adjudicator record a forced choice of negative/positive. Code the final result as the proportion (over adjudicators) of positives.
Have each adjudicator record a forced choice of negative/positive. Code the final result as negative/positive depending on a majority rule. One would need to have an odd number of reviewers for this rule.

When one has a probability of being in a clinical class and such probabilities are not all near 0 or 1, the probabilities are self-contained in terms of capturing the difficulty of the task of classifying patients. This translates directly to quantifying the arbitrariness of forced-choice classifications.

Resources

Probabilistic readjudication of heart failure hospitalization events in the PARAGON-HF study by GM Felker, J Butler, JL Januzzi, AS Desai, JJV McMurray, SD Solomon (includes a multiple imputation approach)
A comparison of approaches for adjudicating outcomes in clinical trials by BC Kahan, B Feagan, V Jairath
Descriptive approach to analyzing observer variability
How breaking ties in a variable increases statistical power
Against diagnosis in favor of matters of degree, by AJ Vickers, E Basch, MW Kattan
Probabilistic prediction in patient management and cinical trials by DJ Spiegelhalter
The end of the “syndrome” in critical care by Lawrence Lynn

Reuse

CC BY 4.0

The Burden of Demonstrating Statistical Validity of Clusters

Frank Harrell — Sun, 06 Oct 2024 05:00:00 GMT

Background

Clustering of patients to find new “phenotypes” is now a fad. For example, repeating the false assertion that diabetes was ever a binary diagnosis, Ahlqvist et al claimed to have found 5 diabetes subtypes using a purely statistical analysis not driven by clinical knowledge. What they found is likely just inefficient prognostic stratification that could be improved upon by directly relating patient characteristics to outcomes.

Maarten van Smeden showed that clustering algorithms easily get the wrong number of clusters when the true number of clusters is known, and Darren Dahly showed in a simple example that clustering is essentially telling us, for example, that people who are older than 65 are older than people who are under 65. van Smeden, Harrell, and Dahly wrote a letter to the editor concerning the Ahlqvist paper, casting extreme doubt on the original authors’ assertions that “new forms of diabetes” have been identified or that this is a useful “step towards precision medicine in diabetes”. van Smeden et al pointed out that direct modeling of outcomes is likely to have much greater payoff, and that the clusters found by Ahlqvist et al are very unlikely to be what they seem. Ahlqvist et al did not even assess within-cluster homogeneity of the component variables nor did they assess within-cluster outcome homogeneity.

See Dennis et al for a direct comparison of Ahlqvist et al’s approach with direct predictive modeling, showing superiority of the latter.

What is the Question and Why Cluster Patients?

Most medical applications of statistical clustering techniques fail to address the most basic questions such as

What is the ultimate goal to which the results of the statistical analysis will be used?
Is the disease being studied all-or-nothing as assumed by clustering algorithms when doing the analysis on “diseased” patients?
What is the best way to summarize the result? Is it patient cluster membership, a clinical prediction model (which much better handles categorical patient characteristics), or is it variable clustering?

Variable clustering often is more likely to meet investigators’ goals than patient clustering. Variable clustering does not discard nearly as much information as patient clustering, is less arbitrary, scales to more variables, and better handles collinearities / redundancies. Sparse principal components analysis (PCA) is also a very useful tool, combining variable clustering with PCA to handle collinearities while providing a more sparse representation of the patient baseline variables. Both of these variable clustering approaches can easiily feed their results into standard clinical prediction models to learn how various dimensions of the patient relate to outcomes.

How Should Clustering Results be Presented?

In the minority of cases where patient clustering is most likely to meet clinical goals, investigators must be made aware that forced-choice classification (assigning each patient to a cluster with no gray zone) is not often the best way to represent clusters. Assignment to discrete clusters assumes that R. A. Fisher’s definition of clusters as compact sets is in play. In other words, pretending that clusters are discrete assumes that clusters are compact, i.e., there is no meaningful heterogeneity within a cluster. When, for example, a patient at the outer boundary of one cluster is closer to a patient at the outer boundary of a different cluster than she is to the center of her own cluster, simply labeling her as a member of “her” cluster is misleading.

It is much more natural to use the results of patient clustering in a continuous, less assumption-laden fashion. For example, one can summarize the results of clustering in the following ways when relating clusters to outcomes:

For clusters and for each patient, compute the distances from the cluster centers as outcome predictors.
Similarly, compute the probabilities that the patient belongs to each of the clusters and use the logits of these probabilities as predictors.

Forced-Choice Cluster Classification Requires Verifying Adequacy of Mere Cluster Membership

When probabilities of cluster membership are not all near 0 or 1, the clusters are not compact enough to be used in forced-choice cluster classification, and likewise if the distributions of distances from cluster centers are wide.

Consider computing the median distance between all possible pairs of cluster centers, and show that the individual patient distances from their own cluster centers is below, say 1/5th of the median distance between cluster centers more than 4/5 of the time to demonstrate that cluster membership is not far from an all-or-nothing phenomenon.

If forced-choice cluster assignments are still of interest, these assignments must be validated with regard to adequacy of summarization of statistical information contained in the original component variables. In other words, demonstrate that the cluster identifiers are sufficient for conveying the information (e.g., phenotypes) the clusters are purported to contain, when there is an outcome or response variable that the clusters are supposed to predict. Here are some useful steps in that endeavor:

Define as the set of indicator variables for membership in clusters
Define as the set of distances a patient has from each of the cluster centers
Fit models to predict patient outcome, with the models containing as predictors both sets and , and models containing and separately
Compute likelihood ratio tests to assess the prognostic information due to each set
Compute the proportion of overall likelihood ratio for & combined that is due to each of the sets
Verify that the proportion of predictive information provided by , after adjusting for , is small. See this and this for more information.
Demonstrate that the clusters provide new prognostic information after accounting for previously known prognostic variables. In a similar fashion to the previous demonstration, replace set with known prognostic variables and compute the fraction of new prognostic information that is provided by the cluster indicators.
Demonstrate that cluster assignments cannot be easily predicted from simple features, using for example polytomous (multinomial) logistic regression.

Demonstrating Stability

Besides adequacy of statistical summarization of component variables, clusters must be validated for stability. A simple bootstrap procedure can document stability of found clusters, and when the number of clusters was not completely pre-specified (before analyzing the data), the number of clusters should be allowed to “float” across resamples, and the frequency distribution of the number of found clusters provided.

Ultimate Validations

Statistical validations of cluster structure and especially of the adequacy of the cluster summarizations are easy. But the clusters then need to be validated in the more difficult way, by demonstrating clinical usefulness of the clusters. Examples of clinical usefulness include

demonstrating that the clusters are clinically interpretable and that patients are homogeneous within the finest level of detail used to summarize clusters
- If forced-choice classification is used, show that there is no remaining clinical information within each choice.
- If distance from all cluster centers are used, show that there is no remaining clinical information once the distance is fixed.
demonstrating within a randomized clinical trial that the clusters are uniquely useful for capturing differential treatment effect, e.g., showing that there is an important interaction between treatment and clusters but no important interaction between treatment and pre-specified raw baseline variables
showing that the number of clusters is clinically correct.

Do similar likelihood ratio assessments as above to compare the total treatment cluster interaction effect to the total treatment cluster distance effects to the total treatment original variable effects. Forced-choice clusters will be embarrassed if the log-likelihood accounted for by simple raw variable (or cluster distance) interactions exceeds that accounted for by cluster memberships.

Reuse

CC BY 4.0

Hosting Web Content

Frank Harrell — Sun, 29 Sep 2024 05:00:00 GMT

One of my best decisions was to build my own web sites hbiostat.org and fharrell.com so that I have total control of content and formatting and can easily and quickly post content updates. I want to share a few things I’ve learned.

While your organization’s web pages are great for static content, my public-facing content evolves rapidly with constant improvements made to course web pages, miscellaneous web pages such as hbiostat.org/data, blog articles, and course handouts. To make it easy to update and to add new pages I’ve found it productive to take control of the situation using web sites that are served by the amazing netlify.com. Some people prefer to create web sites with Github but I like to have total control of formats.

With the Netlify approach you create a static web site on your local computer (I have my main one under ~/web for hbiostat.org), and whenever there is a significant change to your material, you have Netlify re-deploy the web site to Netlify (it only sends what has been changed). I use my own domains but you can use free *.netlify.net domains. Your local computer provides a mirror of what is available publicly and you can easily preview changes by just opening one of your .html files. Deployment can be handled interactively or better by using the Netlify command line app – you just run a single command which I abbreviate as e.g. hdeploy after doing a one-time authentication. If you ever decide to quit using Netlify you have all the web content locally for easy deployment to AWS or anywhere else you want to put it.

I have the deluxe Netlify paid plan because some pages have high traffic but you can do a lot with the Netlify free plan.hdeploy stands for netlify deploy -p -s hbiostat -d ~/web.

To create web content locally I recommend one of the following (or a mix of them):

Simple markdown .md files that are converted to .html using pandoc (creates very small and fast html)
R markdown run through R to create html
Quarto (produces the most beautiful output but if you want the html file to be self-contained and not part of a whole Quarto web site it will create larger html files)

The beauty of Quarto is that it creates books, individual reports, blogs, presentations, and whole web sites. fharrell.com was created completely by Quarto. When you have connected web pages (book chapters; blog articles) the individual html files are lean. Here is an example of a standalone Quarto web page: hbiostat.org/r/hmisc.

The hbiostat.org home page was created using R markdown. The R markdown script is here.

Here is a course web page which I converted from a wiki (see below for a conversion script) to a simple markdown file: hbiostat.org/b2 . The markdown for this is at hbiostat.org/b2/index.md . The pandoc command to convert from .md to .html is

  pandoc --toc --css=https://bootswatch.com/5/cerulean/bootstrap.min.css \
    -s -o index.html index.md

An example of course notes created with Quarto is hbiostat.org/rmsc.

One other lesson I’ve learned over the years after hosting my web pages on Amazon Web Services (AWS) is that when you have to support your own Linux or Windows web server such as an AWS Lightsail instance, the time spent in keeping the site secure and software updated is significant, and doing updates to web pages is not as easy as the local ~/web Netlify mirroring approach. It is far easier to host a static web site where Netlify takes care of 100% of system and web server software issues. There is nothing to update on your site other than the actual web content.

Miscellaneous Tips

Creating an `index` File

Sometimes you want to add a directory full of files to a public web page without taking the time to create index.md to point to each file. The following shell script uses the wonderful tree Linux/Mac app to create index.html such as the one appearing here. Dates are file-last-modified dates.

#! /bin/bash
# Stored in ~/bin/mkindexd

tree --ignore-case -C -I '*confidential*|*cache*|*courseregistrants*' \
  --timefmt "%F  " \
  -H . | sed -e "s/\(.*\)
/\\2\ \ \ 
/" \
  -e "s/\[  \]  //" > index.html

Converting Wiki Content to Markdown

Here is a shell script that converts legacy wiki markdown to regular markdown.

#! /bin/bash
#
# Convert from foswiki-type wiki markup to markdown
# Run e.g. wiki2md foo to convert foo.wiki to foo.md
cat $1.wiki | sed -E -e "s/\[\[(\S+?)\]\[(.+?)\]\]/[\2](\1)/g" \
-e "s/\[\[http(\S+?) (.+?)\]\]/[\2](http\1)/g" \
-e "s/^---+++/###/g" \
-e "s/^---++/##/g" \
-e "s/^---+/#/g" \
-e "s/%N%//g" > $1.md

Reuse

CC BY 4.0

Statistical Thinking

Bayesian Thinking

Modernizing Clinical Trial Design and Analysis to Improve Efficiency & Flexibility

Important Considerations in Trial Design for Flexibility & Efficiency

Parametric Models vs. Empirical Distributions and Ordinal Regression

Background

The Semi-Nonparametric (SNP) Density Family

Simulating Accuracy of for Population Median

Statistical Models Answer the Fundamental Clinical Question and Provide Clinical Trial Estimands

Background

Estimands

Statistical Models

RCT design

Conceptualizing Clinical Questions in an RCT

The Fundamental Question in an RCT

Statistical Estimands to Answer the Fundamental Clinical Question

General Model and Its Underlying Model Parameters

Clinical Scale Treatment Effectiveness Estimands

Celebrate Incorrect Statistical Models

Missing Covariates

Missing Covariates and Mismodeled Measured Covariates

Summary

Further Reading

Reuse

The Unifying Capabilities of Cumulative Probability Semiparametric Models

Background

Link Functions

General Advantages of CPMs

Special Cases

Extensions for Longitudinal Data

Summary

Resources

Reuse

Implications of the Draft FDA Bayesian Guidance

Thoughts About the Roles of AI for Statistics

Questions We Forget To Ask When Designing an RCT

Causal by Design

Background

Study Design

Implications of Study Design on Causal Inference

Summary

Reuse

Goal-Driven Flexible Bayesian Design

Measures of Central Tendency for an Asymmetric Distribution, and Confidence Intervals

Measures of Central Tendency

Confidence Intervals

Distribution and True Location Values

Distributions of Sample Location Estimates

CIs Computed for One Sample

Simulation of CI Coverage

Summary

Confidence Intervals for the Mean Under Skewness: Why Every Fix We Tried Hit the Same Wall

Starting point

The test that broke it

What we tried, and why each attempt failed

The common thread

The resolution: change the estimand, not the method

Bottom line

References

Computing Environment

References

Reuse

Bootstrap Confidence Limits for Bootstrap Overfitting-Corrected Model Performance

Background

Fast Alternative Bootstrap Confidence Limits

Simulation to Derive ABCLOC to Obtain Accurate Confidence Coverage for Simple Performance Measures

Accuracy of Estimated Calibration Curves and Confidence Limits

Accuracy of Bootstrap Calibration Estimates for

Accuracy of Approximate Bootstrap Confidence Limits for Calibration,

Calibration for and

Summary

Computing Environment

References

Reuse

Minimal-Assumption Estimation of Survival Probability vs. a Continuous Variable

Background

Estimation Methods

Simulation

Recommendations

Computing Environment

Re-Write of `lrm.fit` and `orm.fit`

Check Accuracy Against Old `lrm.fit` For a Variety of Levels of Y

Check Speed of `NR`, `LM`, `nlminb`, and `glm.fit`

Check Impact of `initglm` and `transx`

`lrm.fit` vs. `orm.fit` as k

Creating an `index` File