The problem
Across the full 2008–2021 cohort, 11.3% of interventional trials that finished as Completed or Terminated ended Terminated (11.2% in the 2020–2021 test years). An early termination ties up sponsor capital, site staff and IRB capacity. Low accrual is widely reported as a leading reason, but this project has not yet analyzed stop reasons (why_stopped), so reasons are not a result here.
The question is a feasibility review: how much termination risk is visible in the registration record before the first patient is enrolled? The output is a screening and prioritization signal for sponsor feasibility checks, site and network planning, and portfolio triage. It is not a causal model, and it does not say that changing a design parameter would change the outcome.
Results so far
| Model | Role | Test AUC (95% interval) | Test PR-AUC | Precision in top 5% | Validation AUC |
|---|---|---|---|---|---|
| XGBoost, tabular + SVD 200 | Final model | 0.777 (0.767 to 0.785) | 0.306 | 40.7% | 0.735 |
| PyTorch MLP, tabular + SVD 200 | Candidate | 0.780 (0.772 to 0.788) | 0.305 | 39.3% | 0.740 |
| XGBoost, tabular only | Baseline | 0.752 (0.743 to 0.761) | 0.287 | 38.4% | 0.704 |
All numbers are on the test years (2020 to 2021), which were not used to fit or tune any model. AUC measures how well the ranking separates terminated from completed trials: 0.5 is a coin flip, 1.0 is perfect. The final model is the shaded row: the 5-seed averaged XGBoost ensemble on tabular + SVD 200 features. XGBoost and MLP rows are 5-seed averages. Differences and intervals are computed from unrounded estimates, so they can differ by 0.001 from the rounded values shown.
Reference runs: text alone, TF-IDF alone, rank blend
| Run | Test AUC |
|---|---|
| Rank blend: XGBoost + MLP | 0.782 |
| Text features alone (XGBoost) | 0.764 |
| Logistic regression on TF-IDF alone | 0.754 |
Single reference runs without intervals or top-5% precision. The blend averages XGBoost and MLP score ranks and adds +0.005 AUC over the final model (95% interval +0.004 to +0.007).
Text lift. Adding the eligibility-text components raises XGBoost's test AUC from 0.752 to 0.777 (+0.024, 95% CI +0.019 to +0.030).
Calibration. Training up-weights terminated trials, so raw scores are not population probabilities: they average 0.365 against an observed rate of 0.112. A Platt map fit on the validation years (2018–2019) and evaluated on the 2020–2021 test years brings the average to 0.117. Calibration error (ECE, decile bins) falls from 0.253 to 0.018, Brier score from 0.162 to 0.089 (always predicting the base rate scores 0.100), and log loss from 0.493 to 0.302. The ranking is unchanged. The dashboard shows the reliability plot.
Findings
- Ranking power
- The 5% of test trials the final model ranks riskiest (1,559 trials) are 40.7% terminated, 3.6 times the 11.2% base rate. That is precision. Those flagged trials hold 635 of the 3,505 terminated test trials, a recall of 18.1%.
- Text lift is stable
- It is +0.0215 ± 0.0013 across 5 XGBoost seeds (the text model wins all five) and +0.024 for the seed-averaged ensemble. It holds after excluding trials that mention COVID (+0.023) and trials with amendment-style wording (+0.021).
- Eligibility text adds predictive signal
- Text alone (0.764) scores above the structured design fields (0.752), and a plain logistic regression on TF-IDF reaches 0.754, so the gain does not depend on a complex model.
- Few components are needed
- 25 SVD components already give +0.020 AUC, and the gain saturates near k = 100. Validation AUC is flat from k = 100 to 400, so k = 200 is a tie-break, not a sharp optimum.
- Text partly stands in for who is studied
- Oncology and cardiac language raises risk; vaccine and healthy-volunteer language lowers it, so the lift is not eligibility design alone. The text components carry a large share of model influence, spread thinly across 200 columns.
- Why XGBoost is the final model
- The MLP’s AUC is higher (0.780 vs 0.777; paired difference +0.004, 95% interval +0.001 to +0.007): distinguishable in the paired comparison, but small. That edge did not carry over to the top of the ranking: precision in the top 5% is 40.7% for XGBoost and 39.3% for the MLP (one standard error is about 1.2 points). With results this close, XGBoost wins on operations: exact TreeSHAP attributions, no PyTorch dependency in the container, and native handling of missing values.
- The blend
- Averaging XGBoost and MLP ranks adds +0.005 AUC over the final model (95% interval +0.004 to +0.007). It is not planned for serving, so the container needs no PyTorch.
Subgroups
The model flags the 5% highest-scoring trials across the entire test set, one global cutoff (1,559 trials, 40.7% terminated). The table breaks that single shortlist down by group; it does not pick the top 5% separately within each group. Intervals are 95% Wilson intervals on the flagged trials with the cutoff held fixed, so they ignore uncertainty from choosing the cutoff and treat trials as independent. The last column divides precision by the group’s own termination rate.
- The shortlist catches 18% of terminations
- The 1,559 flagged trials contain 635 of the 3,505 terminated test trials, a recall of 18.1%.
- Flags concentrate where termination is common
- Phase 2 trials hold 60% of the flagged trials (939 of 1,559) and treatment trials 87% (1,359 of 1,559), because those groups terminate more often and a single cutoff flags more of them.
- Precision is lower where termination is rare
- Trials with no phase (mostly behavioral and device studies) reach 24.7% (19.1% to 31.4%), below the overall 40.7%. They terminate only 6.0% of the time, so the shortlist is still 4.1 times their own rate: a high lift does not mean high precision when the base rate is low. Industry trials (45.5%, 41.5% to 49.5%) sit above other sponsors (37.6%, 34.6% to 40.8%); the table describes that difference but does not isolate why.
- Within-group ranking is weaker than the overall AUC
- In the notebook 08 refit (overall AUC 0.778), AUC is about 0.68 for Phase 2, 0.71 to 0.72 for Phase 3, Phase 4 and no-phase trials, and about 0.79 for Phase 1. Treatment trials match the overall figure (about 0.78); other purposes range from about 0.60 to 0.75, with wide intervals where few trials terminated. Part of the overall AUC comes from separating groups with different termination rates, so ranking within a group is the harder test. Among phases, Phase 2 has the highest precision but the lowest AUC, which fits its high base rate (26.0%).
- Small groups are uninformative
- Where only 24 to 84 trials are flagged (NIH, diagnostic, prevention, supportive care and other purposes), intervals span 15% to 69%. These estimates do not support a claim that performance is similar or different across those groups.
| Group | Test trials | Flagged | Group termination rate | Precision in top 5% (95% interval) | Lift over group rate |
|---|---|---|---|---|---|
| Trial phase | |||||
| Phase 2 (incl. Phase 1/2) | 4,239 | 939 | 26.0% | 43.9% (40.7% to 47.1%) | 1.7× |
| Phase 1 (incl. Early Phase 1) | 3,911 | 279 | 14.8% | 41.9% (36.3% to 47.8%) | 2.8× |
| Phase 3 (incl. Phase 2/3) | 2,460 | 110 | 19.6% | 40.9% (32.2% to 50.3%) | 2.1× |
| Phase 4 | 1,636 | 45 | 12.4% | 33.3% (21.4% to 47.9%) | 2.7× |
| Not applicable | 18,906 | 186 | 6.0% | 24.7% (19.1% to 31.4%) | 4.1× |
| Lead sponsor class | |||||
| Industry | 8,352 | 585 | 17.4% | 45.5% (41.5% to 49.5%) | 2.6× |
| Other (incl. academic) | 21,628 | 946 | 9.0% | 37.6% (34.6% to 40.8%) | 4.2× |
| NIH | 143 | 24 | 24.5% | 50.0% (31.4% to 68.6%) | 2.0× |
| Primary purpose | |||||
| Treatment | 17,725 | 1,359 | 14.7% | 42.2% (39.6% to 44.9%) | 2.9× |
| Diagnostic | 1,323 | 84 | 11.4% | 34.5% (25.2% to 45.2%) | 3.0× |
| Other | 2,837 | 32 | 6.9% | 34.4% (20.4% to 51.7%) | 5.0× |
| Prevention | 3,510 | 32 | 6.5% | 31.3% (18.0% to 48.6%) | 4.8× |
| Supportive care | 2,367 | 34 | 5.6% | 26.5% (14.6% to 43.1%) | 4.7× |
Groups with fewer than 20 flagged trials are not shown (Missing phase; Other government, Federal, Network, Individual and Unknown sponsors; Basic science, Health services research, Screening and Device feasibility purposes). Rare phase levels are merged into the nearest main phase.
Precision figures use the final model’s saved predictions, as in the dashboard. AUC values come from a separate refit in notebook 08, with bootstrap intervals that resample terminated and completed trials separately within each group, and are read from its plot, so treat them as approximate. These results describe where the ranking works, not why trials terminate.
Validation
Validation mirrors time, not a random shuffle
Label: Completed is negative and Terminated positive. Withdrawn trials (stopped before enrolling anyone) are excluded and analyzed separately. Unknown-status trials are bounded under extreme assumptions rather than ignored; the bounds are wide.
Leakage control
AACT is a latest-snapshot database, not a protocol-time record, so the tables mix registration-time fields with fields written after the outcome. The model uses fields selected to approximate registration-time information: everything obviously written later is excluded. That is a field choice, not a verified day-one record, and it appears below as an open caveat.
| Field group | Decision | Why | Evidence |
|---|---|---|---|
Actual enrollment (enrollment_type = ACTUAL) | Excluded | Realized enrollment can reflect information accumulated after registration, so it is not treated as a registration-time feature | Value audit; estimated enrollment also dropped |
why_stopped | Excluded | Written at termination | Not a predictor. Used only for label validation; the stop-reason error analysis is pending |
| Completion dates, results fields | Excluded | Written after the outcome | Excluded by construction |
| Last-update date, amendment and version counts | Excluded | Record recency tracks trial age and outcome | Excluded by construction |
| FDA-regulated drug and device flags | Excluded | Dropped from the main model | n/a |
| Site, country and condition counts | Ablation only | Accumulate after registration | Test AUC 0.7513 to 0.7493 when added: no gain |
healthy_volunteers | Used, checked | Registered eligibility flag; the top feature by SHAP in the baseline (about 21% of total attribution) | Dropping it costs 0.007 test AUC (0.7513 to 0.7442) and 0.004 PR-AUC; the termination gap (4.5% vs 14.8%) is stable across splits and years and holds within every phase; sponsor mix is the same for both values. Baseline, single fit |
| Remaining structured design fields | Used | Phase, sponsor class, allocation, masking and similar design choices | Baseline test AUC 0.752 |
| Eligibility text | Used | Part of the protocol record; transforms fit on the training period only | +0.024 AUC (CI +0.019 to +0.030); partly a therapeutic-area proxy |
| AACT is a latest-snapshot database | Open caveat | Kept fields may have been edited after registration | Mitigated by field choice; not checked against historical versions |
Flat metrics after adding the count features show they add nothing here. They do not prove the remaining features are leak-free. For healthy_volunteers, the strongest baseline feature, the checks above show a stable relationship and a modest cost of removing it, which argues against drift or a post-outcome edit but cannot rule out an edit to the latest AACT record. The drop-and-compare used the structured baseline, not the final text model.
Limits
- Registry data is sponsor-reported and unevenly kept up to date.
- Trials with Unknown status are excluded from training; if that missingness tracks termination risk, estimates lean optimistic.
- Associations only. Nothing here supports a causal claim about design choices.
- Registration-time describes the field choice, not a verified day-one record, because AACT keeps only the latest version.
- Subgroup estimates are noisy where few trials are flagged (see the Subgroups section).
- Open checks: error analysis by stop-reason (
why_stopped) has not started, and thehealthy_volunteersdrop-and-compare was a single fit on the structured baseline, not yet repeated on the final text model across seeds. - The API and pretrained-embedding experiments are planned and are not part of these results. Workflow and project status are in the repository README.
- The test window overlaps the COVID era; performance on later years is untested, and there is no retraining loop because there is no production label feedback.