Thanh M. Brown Data Analyst · Machine Learning & Analytics
← Back to projects

02 — Clinical ML · Epidemiology · Matched Case-Control

Osteoporosis Risk Classification & Recall Asymmetry

Why do models miss osteoporosis cases on a perfectly balanced dataset? Across 61,026 matched patients, classifiers reach ROC-AUCs up to 0.79 yet miss 40–49% of cases. I validated the cohort by reproducing its source study's published odds ratio, then investigated where the gap comes from.

  • Matched case-control
  • Conditional logistic regression
  • Multiple imputation
  • Model diagnostics
  • Threshold & recall analysis
  • Subgroup analysis
GitHub repository
61,026
matched patients, 1:1 by design
0.79
top ROC-AUC (XGBoost)
40–49%
of cases missed at the 0.5 cutoff
3.98
reproduced odds ratio vs. 3.97 published

The puzzle

The data are the matched case-control cohort from Usala et al. (2015), rebuilt from the underlying EHR records. The cohort is exactly 50/50 cases and controls by construction. Even so, every model recalls controls far better than cases at the default 0.5 cutoff.

Out-of-fold predictions, 10-fold cross-validation split by matched pair, full-record lab columns excluded.

For screening, a strong ranking score does not mean a model catches most patients with the disease. Where the threshold sits, and who falls just below it, decides that.

The evidence

All four models separate cases from controls well above chance.

ROC curves for the four models: XGBoost AUC 0.790, random forest 0.789, logistic regression 0.749, linear SVM 0.738, all well above the diagonal.
ROC curves, out-of-fold, full-record lab columns excluded.

But cases are bimodal, and one of their peaks overlaps the controls just under the 0.5 cutoff.

Histograms of out-of-fold predicted probability by true class for four models. Controls pile up around 0.35 to 0.40. A large share of cases piles up in the same range, and a second case peak sits near 0.8 to 1.0.
Predicted probability by true class; the dashed line is the 0.5 cutoff. Controls are largely unimodal, clustered around 0.35–0.40. Cases are bimodal: one peak overlaps the controls, the other sits high. Tree ensembles (Random Forest and XGBoost) push that high-signal group all the way to 1.0, while logistic regression and linear SVM place it lower, around 0.9 and 0.8.

The same distributions, viewed through their medians: controls sit tightly below the cutoff, while the typical case sits barely above it.

Violin plots of the same predicted probabilities. Control medians sit around 0.36 to 0.40 in a narrow band below 0.5. Case medians sit around 0.52 to 0.59, with scores spread across the full range.
Same out-of-fold predictions as the histograms above, shown as violin plots with box plots. Control medians fall around 0.36–0.40; case medians fall around 0.52–0.59, with case scores spread across the full range. Pooled across both classes, the median prediction is only 0.39–0.43, so the models label just 34–39% of patients positive on a 50/50 cohort.

Validation

Before any modeling, I rebuilt the source study (Usala et al., 2015) and reproduced its hyponatremia–osteoporosis association with conditional logistic regression stratified on the matched pairs, pooling five BMI imputations by Rubin's rules.

3.98reproduced (95% CI 3.60–4.40)
3.97published
Imputation check
BMI was missing for about 68% of patients, far more often among cases, yet adjusting for it barely moved the association, so the reproduction is not driven by the imputation.

Safeguards

Matched observational EHR data breaks assumptions that standard ML pipelines take for granted.

Temporal leakage
Full-record lab summaries include post-diagnosis measurements (sodium is recorded for 99.6% of patients across the full record vs. 67.4% before the index date), so all full-record features were excluded.
Pair-grouped cross-validation
Folds are assigned by matched stratum, so a case and its matched control never fall on opposite sides of a split.

Finding

Not class imbalance
Exact 50/50 balance rules it out by design.
Not a single algorithm
The gap persists from linear models to gradient boosting. Moving from logistic regression to XGBoost lifts AUC from 0.749 to 0.790 but only narrows the gap from 0.29 to 0.24.
Ranking is not the problem
Tree ensembles rank a case above its matched control about 79% of the time, well above chance.
Case heterogeneity
The evidence so far points to a high-signal case subgroup the models catch, alongside a second group that overlaps heavily with controls.

Stage 3 is in progress: separating missingness flags from lab deciles and computing SHAP values for the two confidence groups.

Tech Stack

  • LanguagePython
  • Modelingscikit-learn · XGBoost
  • Statisticsstatsmodels
  • MethodsConditional logit · Rubin's rules

Code, notebooks, and full documentation are in the repository. Patient data is institutional and not redistributed.

Read the full analysis on GitHub