Rural hospital closure risk — held-out evaluation

Calibration integrity

Checked before anything else, because a broken calibrator silently invalidates every number below it.

Degenerate calibration detected

XGBoost produced a calibration map that collapsed the model's ranking. A near-constant predictor scores AUROC 0.5000 by construction, so the hold-out metrics for the affected models below measure the calibrator, not the model. This page was generated from artifacts that still carry the defect — re-run python run.py train evaluate and regenerate. The banner is the guard in tests/test_calibration.py doing its job, not a rendering error.

Isotonic maps are monotone, so they must not reorder scores and must keep enough distinct values to threshold on. Rank correlation below 0.95, or fewer than 10 distinct values, fails the guard in `src/modeling/calibration.py`.
Model	Distinct values	Rank corr. vs raw	Share at mode	Verdict
XGBoost	2	0.016	100.0%	collapsed
L2-penalized logistic	27	0.995	15.3%	preserved
Discrete-time hazard	17	0.988	25.0%	preserved

Hold-out performance

PR-AUC is the primary metric. At this base rate AUROC flatters a model that ranks a handful of positives well while flagging thousands of negatives.

One-time evaluation on 2019–2021. Hyperparameters were selected on ≤2018 folds only, asserted in `tests/test_holdout_boundary.py`.
Model	CV PR-AUC	Hold-out PR-AUC	AUROC	Brier	Lift over base
Discrete-time hazard (survival)	0.0218	0.0011	0.6833	0.000530	2.2×
XGBoost (gradient-boosted trees)	0.1313	0.0005	0.5000	0.000516	1.0×
L2-penalized logistic	0.0365	0.0009	0.7050	0.000569	1.8×

An AUROC of exactly 0.5000 is highlighted. It is not a result — it is what a constant predictor returns, and it means the calibration audit above should be read first.

Where the real closures ranked

The question a summary metric cannot answer: of the 6 hospitals that actually closed, how near the top of the list did the model put them?

XGBoost

CCN	Year	Region	Rank	Percentile
140143	2021	Midwest	2 of 11,760	100.0
160008	2020	Midwest	2 of 11,760	100.0
170074	2019	Midwest	2 of 11,760	100.0
340133	2021	South	2 of 11,760	100.0
390071	2021	Northeast	2 of 11,760	100.0
490098	2019	South	2 of 11,760	100.0

L2-penalized logistic

CCN	Year	Region	Rank	Percentile
340133	2021	South	1,433 of 11,760	87.8
390071	2021	Northeast	1,433 of 11,760	87.8
490098	2019	South	3,117 of 11,760	73.5
170074	2019	Midwest	3,117 of 11,760	73.5
140143	2021	Midwest	3,667 of 11,760	68.8
160008	2020	Midwest	4,787 of 11,760	59.3

Discrete-time hazard

CCN	Year	Region	Rank	Percentile
140143	2021	Midwest	2,120 of 11,760	82.0
170074	2019	Midwest	2,120 of 11,760	82.0
340133	2021	South	2,120 of 11,760	82.0
390071	2021	Northeast	2,120 of 11,760	82.0
490098	2019	South	2,120 of 11,760	82.0
160008	2020	Midwest	5,172 of 11,760	56.0

Ranks, not scores. A calibrated probability of 0.4% sounds like nothing until you note the base rate is 0.051%.

Score distribution

40 bins, counts on a log scale so a single spike does not flatten the rest of the shape.

XGBoost

0.000% 25.000%

median: 0.000%
95th: 0.000%
distinct: 2

L2-penalized logistic

0.000% 3.586%

median: 0.209%
95th: 0.877%
distinct: 27

Discrete-time hazard

0.000% 5.056%

median: 0.130%
95th: 0.603%
distinct: 17

Diagnostics

Interpretability, calibration, net benefit, and fairness slices.

SHAP summary plot ranking feature contributions for the XGBoost model — Feature contributions, XGBoost. Operating margin and occupancy rate lead, which matches the rural-distress literature.

Reliability diagram plotting predicted probability against observed frequency — Reliability after isotonic calibration. Deviation from the diagonal at the high-score end is expected with six positive events.

Decision curve plotting net benefit against threshold probability — Decision-curve analysis. Net benefit against a screening threshold, versus flagging everyone and flagging no one.

Bar chart of model performance broken out by US Census region — Performance by Census region.

Bar chart of model performance broken out by hospital ownership category — Performance by ownership category.

Bar chart of model performance broken out by hospital bed-size tier — Performance by bed-size tier. Read all three cautiously: subgroup counts are tiny.

How to read this

Sample size. 6 positive events in 11,760 rows. Every point estimate here carries wide uncertainty and none should be quoted without that qualifier.

Policy shock. The hold-out overlaps the pandemic. CARES Act and Provider Relief Fund transfers altered hospital finances in ways no pre-2019 training data encodes.

Screening, not prediction. Scores rank hospital-years by relative risk. They are not probabilities anyone should act on, and the model card lists the out-of-scope uses explicitly.

Historical, not current. All scores cover 1997–2021 and are published for reproducibility. They say nothing about any hospital's condition today.

Identifiers. Keyed by CMS Certification Number, a public identifier. Every input is public CMS data — HCRIS cost reports and the UNC Sheps Center closure list. No PII beyond public records.

Generated by python run.py dashboard (src/scoring/dashboard.py) from the artifacts in reports/modeling/. No JavaScript and no tracking. The only external request is a webfont from Google Fonts; the page renders correctly in a system serif without it.