Four models. 1.3 million patient records across two population datasets. One central finding: a model achieving 0.794 AUC internally drops to 0.717 when tested on a population 83× larger than it trained on, and predicts elderly patients at only 0.607. Fairness in clinical ML is not optional.
Type 2 diabetes affects over 537 million people worldwide (IDF Atlas, 2021). Early identification of high-risk individuals enables targeted prevention — glycemic control, lifestyle interventions, and reduced complication burden. Machine learning holds genuine promise here, but the published landscape is crowded with single-dataset evaluations, no fairness assessment, and no external validation. Claims of high AUC are published daily; evidence those claims survive deployment is rare.
This paper breaks from that pattern. Four algorithms are trained and evaluated under identical preprocessing and cross-validation conditions, the winning model is validated on a fully external, much larger dataset with zero retraining, and performance is systematically measured across protected subgroups defined by age, BMI, race, and gender. The goal is not to crown a "best" algorithm — it is to characterize where that algorithm breaks down in deployment.
Figure 1 — Study design: train on NHANES (15,685), validate without retraining on BRFSS (1,285,783) — a population 83× larger, surveyed by phone instead of clinical exam.
Both surveys use only non-laboratory, self-reportable predictors — age, sex, race, BMI, smoking status, physical activity, and history of heart attack or stroke — deliberately, so the model could realistically run in a clinic or pharmacy with no blood draw. The huge jump in scale and collection method between training and validation simulates a real deployment: a model built on one health system, handed to a population it has never seen, collected a different way.
Four algorithms — XGBoost, Random Forest, an SVM, and Logistic Regression — were trained identically and compared; XGBoost led, but by a margin too small to matter clinically (full comparison below).
XGBoost achieves 0.794 AUC internally. On the fully external BRFSS cohort — same features, zero retraining, 83× more patients — it drops to 0.717. A 9.7% relative degradation that internal cross-validation cannot detect.
This 0.077 AUC drop represents the real-world distribution shift between a clinically-surveyed training cohort and a much larger, phone-surveyed population. The drop is not catastrophic — 0.717 remains clinically useful — but it is larger than most published benchmarks acknowledge. Papers reporting only internal cross-validation performance are systematically overestimating the clinical value of their models.
Figure 2 — Internal vs external validation. XGBoost's 9.7% relative AUC drop on a population it never saw illustrates why external validation is essential before clinical deployment.
Performance is not uniform across patients. The sharpest gap is age: XGBoost reaches 0.742 AUC for adults 18–39 but only 0.607 for patients ≥60 — precisely the group with the highest diabetes burden and the most to gain from early detection.
Figure 3 — Fairness forest plot: XGBoost AUC by age and BMI subgroup. Elderly patients (≥60) reach only 0.607 — 13.5 pp below young adults (0.742).
All four algorithms trained under identical preprocessing: median imputation for continuous features, one-hot encoding for categoricals, MinMax scaling, SMOTE oversampling for class imbalance. Hyperparameter optimization via 100-iteration Bayesian search. Primary metric: AUROC, with Brier score and Expected Calibration Error (ECE) for calibration quality.
Figure 4 — Internal AUROC comparison. XGBoost leads at 0.794; all four models competitive within 0.018 AUC — suggesting the feature set, not the algorithm, is the binding constraint.
The narrow 0.018 AUC spread across all four models is itself a finding: the performance ceiling is set by the available features, not the algorithm. Any model claiming substantially superior performance on similar data is likely overfitting to training-set idiosyncrasies.
SHAP (SHapley Additive exPlanations) values quantify each feature's average contribution to the model's output. Unlike permutation importance, SHAP values are directional and additive — showing exactly how much each feature moves the prediction for each patient. This makes them interpretable in clinical terms: not "feature X matters" but "feature X pushes this patient's predicted risk up by this amount."
Figure 5 — SHAP feature importance for XGBoost. Age dominates (mean |SHAP| = 0.142), followed by BMI (0.098) and physical activity (0.067). Race/ethnicity ranks 4th at 0.054, making subgroup fairness auditing essential.
The race/ethnicity SHAP value of 0.054 — ranking 4th — means race is the model's 4th most influential input. This is not automatically problematic: race correlates with documented disparities in diabetes prevalence and access to preventive care. But it creates an obligation: a model that uses race as a significant predictor must be audited for disparate impact. The paper documents moderate but real variation in sensitivity and specificity across racial and ethnic groups.
AUC measures discrimination — the ability to rank patients by risk. Calibration measures whether predicted probabilities are accurate in absolute terms: a model predicting 30% risk should see observed events in roughly 30% of such cases. Overall Brier score on external validation is 0.123, with moderate overconfidence (ECE = 0.07) consistent with gradient-boosting behavior on out-of-distribution data; Platt scaling reduces ECE to 0.03 without affecting discrimination.
Clinical decision-making that relies on absolute risk thresholds — e.g., "refer all patients with ≥25% predicted risk" — requires well-calibrated models. Reporting only AUC is insufficient. The paper includes Brier scores and reliability diagrams alongside AUC for all four models, providing the full picture required for clinical deployment decisions.
1. External validation is not optional. Publishing internal-only cross-validation AUC without external validation actively misleads clinicians about deployment performance. Every ML model proposed for clinical use should be tested on a held-out external cohort before publication.
2. Subgroup audits must explicitly cover elderly patients. Standard benchmark datasets overrepresent working-age adults. Models trained on pooled populations will systematically under-identify elderly high-risk patients — precisely the group with the highest benefit from early intervention.
3. Race as a predictor requires monitoring, not deletion. Removing race from features does not eliminate disparate impact — it can worsen it by forcing the model to use correlated proxies. The correct response is explicit subgroup auditing with race-stratified performance metrics.
4. Algorithm selection is second-order. The 0.018 AUC spread across four algorithms implies that clinical ML research should invest in representative data collection and feature engineering rather than algorithm benchmarking. XGBoost leads, but only by a margin indistinguishable in clinical terms from Logistic Regression.
Full title: Comprehensive Evaluation of Machine Learning for Type 2 Diabetes Risk Prediction: Large-Scale External Validation and Fairness Analysis
Venue: Accepted, IEEE, 2025.
Authors: Rajveer Singh Pall, Sameer Yadav (corresponding), Siddharth Bhalerao, Sourabh Sahu, Ritu Ahluwalia, Bhaskar Awadhiya.
Training data: NHANES 2015–2020 (NIH), 15,685 records, 8 non-laboratory predictors, 5-fold stratified CV, SMOTE oversampling.
External validation: BRFSS 2020–2022 (CDC), 1,285,783 records, identical feature mapping, zero retraining.
Models: XGBoost, Random Forest, SVM (RBF kernel), Logistic Regression — all with 100-iteration Bayesian hyperparameter optimization.
Evaluation suite: AUROC (primary), Brier score, Expected Calibration Error (ECE), subgroup-stratified AUC by age/BMI/race/gender, SHAP importance, Platt scaling post-hoc calibration, 1,000-iteration bootstrap CIs.