Evidence
Review model suitability, temporal forecast evaluation, active-model diagnostics, provenance, and concise limitations for the latest committed run.
Current committed benchmark run
This evidence describes the current deterministic synthetic benchmark. It does not validate a future uploaded dataset; governed runtime validation remains pending P1.4.
Validation Design
Expanding-Window Rolling Origins
Each fold trains on eligible historical rows, embargoes the row whose two-week target is not yet available, and evaluates the next weekly origin. Training history then expands by one week.
Historical training data
104-row initial history
One-row label embargo
Unavailable future target excluded
Weekly forecast origin
68 deterministic folds
Unseen target
2 weeks ahead
Expanded history
1-week step
Primary method
expanding window rolling origin
Fold count
68
Initial history
104 rows
Forecast horizon
2 weeks
Step
1 week
Features
18 canonical features
Label policy: target_label_available_when_target_period_is_at_or_before_forecast_origin. Real reporting publication delays and revision vintages are not modeled yet. Results use deterministic synthetic benchmark data and do not establish real-world Dhaka performance.
Candidate comparison status: comparison_complete_and_adopted. Seven fixed candidates use these same fold descriptors; preprocessing is fitted inside each fold.
This recommendation applies to the current synthetic benchmark dataset and does not establish suitability for every future upload.
Selection status
Comparison and active model
Comparison status
Comparison complete — selected model adopted
Comparison winner
Random Forest
Active forecast model
Random Forest
Adopted through governed model review. Random Forest is active for this synthetic benchmark deployment only.
| Candidate | MAE | RMSE | WAPE | Successful / failed | Eligible |
|---|
Selection: Random Forest had the lowest eligible MAE under the declared deterministic rule.
Current synthetic benchmark only; materially different datasets require dataset-specific reassessment.
Uncertainty evidence
Temporally Evaluated Empirical Forecast Range
Each evaluated fold uses only absolute Random Forest residuals from earlier folds. The committed compact projection records 48 historical evaluation folds. Targets overlap and the same fold set informed model selection.
Nominal target
90% empirical
Historical coverage
89.5833%
Covered / evaluated
43 / 48
Interval status
Not a prediction interval
Warm-up folds
20
Average width
146.9243 cases
Lower / upper misses
2 / 3
Current empirical range: 53–187 cases around the 120-case forecast
High-incidence and rising-period performance may be weaker. Historical empirical coverage is not a probability guarantee.
Planning sensitivity scenarios remain separate
Operational compatibility continues to use 87 / 120 / 153. These legacy RF RMSE planning values are uncalibrated and do not represent the empirical forecast range.
Synthetic rolling evidence is not real-world Dhaka calibration. The range is not a probability statement or guarantee.
Active-model evidence
Active Random Forest rolling performance
- MAE
- 31.73
- RMSE
- 49.56
- WAPE
- 26.78
Historical compatibility evidence
Historical P1.1 Gradient Boosting rolling-validation evidence — not active-model performance
Forecast Quality
Actual vs Predicted Dengue Cases
“One unseen two-week-ahead target at each rolling forecast origin.”
Rolling Origins · 68 folds
Source: data/chart_data.json · actual_vs_predicted
Error Analysis
Forecast Error by Model
“Lower is better.”
Primary metrics aggregate all rolling-origin folds; active uncertainty uses RF rolling residuals, while the legacy holdout RMSE remains planning compatibility only.
MAE by Model
Mean Absolute Error — lower is better
RMSE by Model
Root Mean Squared Error — penalizes large errors
Source: data/chart_data.json · model_error_bars
Feature importance is a model diagnostic and does not establish causality, biological mechanism, clinical importance, or stability across seasons. These diagnostics describe the selected Random Forest validation-model instance and do not directly explain the separately fitted all-data forecast instance.
Historical Gradient Boosting evidence
Historical · Compatibility-only · Not active-model evidence.
Technical artifact reference
data/model_explainability.json
Run status
Committed successfully
Active model
Random Forest
Deployment gate
Benchmark only
Technical IDs, artifact identities, and hashes
- Run ID
- 457f334c-7085-4079-a7db-58e6a9ff8c5f
- Manifest SHA-256
- ff2126bccb8a39d5316bc6b4c216cd967e74a90ea81da723756196674f57ba6a
- Formula registry SHA-256
- e67552dbe4b5a80b9a082f0beb3ff70ba0ad5378a25b845e8d195352df3357cc
- Deployment profile SHA-256
- cb53f1a0e015b514cd129c42dc10d2a530b88a2712e1565ab72eae2730c744d8
Known Limitations
Limitations of the Current Validation
The following limitations are acknowledged transparently as part of the prototype validation design.
Rolling-origin validation repeatedly evaluates unseen future periods, but current results are based on deterministic synthetic benchmark data and do not establish real-world Dhaka performance.
Real reporting publication delays and revision vintages are not modeled yet.
Permutation importance was not calculated per fold because each rolling-origin fold contains one validation row, making permutation diagnostics statistically degenerate and unsuitable for temporal-stability claims.
Trained and evaluated on controlled synthetic weekly Dhaka South demonstration data. Zone-level spatial allocation uses a synthetic heuristic, not official sub-district surveillance.
The available synthetic history is insufficient to establish stable generalisation to real outbreak cycles.
No real hospital-level validation — dengue bed capacity, occupancy, and inventory data are illustrative.
No ward-level or sub-zone spatial outcome validation — zone allocation is heuristic, not calibrated.
The empirical range is temporally evaluated on synthetic residuals; it is not a prediction interval or probability guarantee.
The aggregate-MAE winner is result-driven and is not a universal claim about the best dengue forecasting model.
Random Forest has limited extrapolation ability; materially different datasets require a new dataset-specific suitability assessment.
Real deployment would require official surveillance data, clinical validation, and institutional partnerships.
Importance Stability
Native importance stability is aggregated across 68 fitted fold estimators.
Permutation stability: not_evaluated_single_row_folds. P0.4 holdout permutation diagnostics remain available separately and unchanged.
Candidate Comparison Boundary
Models were compared on the same deterministic synthetic rolling-origin folds. The selected model is the best-performing demonstration candidate under the declared rule, not a proven real-world dengue model.
Comparison winner: random_forest; active model: random_forest; adoption: adopted_p1.2b. Forecast uncertainty uses prior-only synthetic temporal residual evaluation; legacy RMSE values remain planning scenarios only.
Interpretation Guidance
What Validation Proves / Does Not Prove
Clear framing for technical evaluators and IEEE reviewers.
What It Proves
- The pipeline supports chronological backtesting with time-based train/test separation.
- Naive and Moving Average baselines are compared systematically against ML output.
- Model errors (MAE, RMSE, MAPE) are quantified and reported transparently.
- Forecast output can be converted into structured uncertainty scenarios.
- The dashboard can surface model evidence to technical evaluators.
What It Does Not Prove
- Does not prove clinical validity or epidemiological accuracy.
- Does not prove official public health deployment readiness.
- Does not validate real facility stock levels or bed capacity.
- Does not claim algorithmic novelty in the forecasting method.
- Does not demonstrate performance on real Dhaka South outbreak data.
This validation layer is included so technical evaluators can inspect model behaviour. Operational users do not need to interpret RMSE or MAE directly; they receive translated preparedness outputs such as SDH, bed gap, and priority directives.
Portfolio / Evaluator Note — DengueOps AI · IEEE ICADHI 2025