Leakage-aware pre-event machine learning evaluation for horse-race prediction under temporal validation.

Overview

• This study developed and assessed a machine learning framework designed to predict horse race outcomes using data available only after race-entry finalization but before race results, emphasizing detection and prevention of data leakage during temporal validation.
• The research used Japanese horse-race data, applied temporal splits for training, validation, and testing, and compared performance between different model feature sets to ensure robustness and generalizability over time.

Introduction and Problem Context

• Horse race prediction is a challenging task often addressed with machine learning models.
• Data leakage, which occurs when information unavailable at prediction time influences model training or evaluation, can lead to overly optimistic performance estimates.
• This study focuses on leakage-aware modeling, ensuring that only information valid before race outcomes is used to build predictive models.
• The timing constraint: using data available after race-entry finalization but before the race outcomes enables practical pre-event prediction without hindsight bias.

Data and Methodology

• Data Source:
  • Japanese Racing Association (JRA) flat-race data sourced from the JRA-VAN Data Lab and exported with TARGET frontier JV.
• Temporal Dataset Splits:
  • Training Set: 2015-2022
  • Validation Set: 2023-2024
  • Independent Test Set: January 5, 2025 to May 10, 2026
  • The test set consisted of 63,910 horse-level observations from 4,556 races, representing a large and realistic independent sample for evaluation.
• Outcome Variables:
  • Two binary outcomes were evaluated:
    • Win outcome — whether a horse won the race
    • JRA place-rule-compatible place outcome — corresponds to placing within a position as determined by the JRA place rules, based on a processed field_size variable
• Performance Measurement:
  • Model discrimination was measured primarily using the Receiver Operating Characteristic Area Under the Curve (ROC AUC).
  • Confidence intervals for performance were estimated with 10,000 race-level bootstrap resamples, a robust statistical resampling method preserving race-level correlation.

Models Compared

• current_full_model: Augmented model that included a domain theory-derived “theory-score” feature block.
• matched_no_theory_model: Model with the same hyperparameters but without the theory-score block, representing a sensitivity analysis to test the added value of the domain-derived features.
• Both models were evaluated to check whether domain knowledge features improved prediction accuracy or not.

Key Results

• For the win outcome:
  • matched_no_theory_model ROC AUC = 0.7543 (95% CI: 0.7475–0.7609)
  • current_full_model ROC AUC = 0.7293 (95% CI: 0.7224–0.7362)
  • The model without the theory block performed better, indicating the theory features did not improve predictive ability.
• For the JRA place-rule-compatible place outcome:
  • matched_no_theory_model ROC AUC = 0.7513 (95% CI: 0.7469–0.7558)
  • current_full_model ROC AUC = 0.7164 (95% CI: 0.7118–0.7212)
  • Again, the no-theory model outperformed the theory-augmented model.
• An exploratory diagnostic aimed at quantifying uncertainty at the race level was assessed but kept separate and not incorporated into horse-level models.

Interpretation and Implications

• The domain theory-derived feature block (“theory-score”) did not improve model performance on out-of-sample temporally separated data.
• This may imply that handcrafted or domain-specific features require rigorous validation to justify their inclusion in predictive models.
• The study emphasizes the importance of:
  • Conducting leakage audits to prevent information leakage through improper feature selection or data handling.
  • Using temporal validation to realistically evaluate model generalizability over time.
  • Performing feature-block ablation studies and matched sensitivity analyses for assessing incremental feature utility.
• These best practices help ensure that deployed machine learning models for real-world scenarios like horse racing remain reliable and robust without bias from data leakage.

Conclusion

• The proposed leakage-aware framework demonstrated a rigorous approach to horse race outcome prediction under realistic temporal constraints.
• Removing potentially leaking or less useful theory-derived features improved validation outcomes, reinforcing data-driven feature selection importance.
• The research supports a systematic approach to pre-event prediction model development, focusing on temporal validation and leakage prevention to ensure accurate generalization.