Submitted · IEEE TNNLS

The CPFE Audit

Cross-platform generalisation failure in mental-health NLP — a five-axis Cross-Platform Fairness Evaluation protocol.

ProblemMental-health text classifiers that test well on one social platform quietly fail when used on another — and one aggregate accuracy score hides it.

ApproachA five-axis audit trains classifiers on one platform and tests them on another, watching accuracy, calibration, fairness, and reasoning all at once.

ResultMoving platforms alone drops accuracy 30-39% and makes the models attend to almost completely different words.

SignificanceA model that passes its own benchmark can still fail the moment real users behave differently than its training data.

30–39%AUC degradation

9×ECE increase

<0.17Minority disparate impact

≈0.10Attribution overlap (Jaccard)

The Problem

Mental-health text classifiers that look strong on one social platform fail systematically when deployed on another — and the failure is hidden: accuracy, calibration, and equity all degrade together, but a single aggregate metric reports none of it.

The CPFE Protocol

Five axes audit a family of four transformers (BERT, RoBERTa, Emotion-DistilRoBERTa, GoEmotions) trained on one platform and tested across Reddit and Twitter: discrimination (macro-OvR AUC with DeLong CIs), calibration (ECE), significance (Bonferroni-corrected bootstrap), equity (symmetric disparate impact — the novel axis), and attribution stability (gradient-saliency Jaccard@K).

Outcome

Under platform shift, AUC falls 30–39%, calibration error rises 9×, minority-class disparate impact drops below 0.17, and token attribution overlap collapses to ≈0.10 — the models attend to entirely different words. The audit makes a failure that aggregate accuracy hides both visible and measurable.

Methodology Pipeline

Key Result

Same platform tested

Stable

Accuracy, calibration, and word-level reasoning all hold up.

→

Cross-platform tested

Collapses

AUC falls 30-39%, calibration error rises 9×, and attribution overlap drops to ≈0.10 — different words entirely.

Same model, same task — only the platform changed

↗ View on GitHub