Help & Documentation

Learn how to use the NFL Simulation Engine. This guide covers everything from basic concepts to advanced workflows.

Overview: What Does This System Do?

The NFL Simulation Engine predicts NFL game outcomes using a Bayesian statistical model. Here's what it does:

• Trains models on historical play-by-play data to learn team strengths, QB effects, and situational factors
• Simulates games thousands of times to generate win probabilities and score distributions
• Backtests model performance on historical games to evaluate accuracy
• Calibrates predictions with betting market data (optional)

Key Concepts

1. EPA (Expected Points Added)

EPA measures how many points a play is worth on average. A +3 EPA play means the team gained about 3 points of expected value. The model predicts EPA for each play based on:

• Team offensive/defensive strength
• QB quality
• Down and distance
• Field position
• Game situation (score, time remaining)

2. Model Training

Training teaches the model patterns from historical data. You specify:

• Seasons: Which years of data to use (e.g., 2016-2024)
• Profile: How thorough the training should be
  • Dev: Quick test (~5 min, 50k samples)
  • Fast: Standard training (~15 min, 120k samples)
  • Full: Comprehensive (~2+ hours, all data)
• Inference Method: How the model learns
  • ADVI: Faster, approximate (good for dev/fast)
  • NUTS: Slower, exact MCMC (best for full training)

3. Simulation

Simulation runs the game thousands of times using the trained model. Each simulation:

1. Generates drives using EPA predictions
2. Converts drive EPA to points
3. Alternates possessions between teams
4. Records final score

After 10,000 simulations, you get distributions: "Chiefs win 58% of the time, average score 27-24."

4. Model Artifacts

Each trained model is saved as an "artifact" with:

• Model Hash: Unique identifier (e.g., "a3f2b1c9")
• Seasons: Training data used
• Metrics: Performance statistics
• Active Status: Whether it's used by default

You can have multiple models and switch between them. Only one can be "active" at a time.

Basic Workflow

Here's the typical workflow for using the system:

Training a Model: Detailed Guide

When to Train

• First time using the system
• After new season data becomes available
• When you want to test different training configurations
• When you want to use different season ranges

Training Parameters Explained

Seasons

Format: Comma-separated list (e.g., "2016,2017,2018,2019,2020,2021,2022,2023,2024")

Recommendation: Use recent seasons (last 5-8 years). Older data may be less relevant due to rule changes.

Training Profile

Inference Method

• Auto: Uses ADVI for dev/fast, NUTS for full (recommended)
• ADVI: Faster, approximate Bayesian inference
• NUTS: Slower, exact MCMC sampling (best quality)

Example: Your First Training

Monitoring Training Progress

After starting training:

1. You'll be redirected to the job detail page
2. Watch the log for progress updates
3. Status will change: Queued → Running → Succeeded
4. When complete, model appears in Model Registry

Running Simulations: Detailed Guide

Required Information

• Home Team: 2-3 letter abbreviation (e.g., "KC", "BUF", "SF")
• Away Team: 2-3 letter abbreviation
• Home QB: Full name (e.g., "Patrick Mahomes")
• Away QB: Full name (e.g., "Josh Allen")

Optional Parameters

• Number of Simulations: Default 10,000 is good. More = more accurate but slower.
• Random Seed: For reproducibility (default 42 is fine)
• Model: Use active model (default) or select specific model
• Market CSV: Path to betting market data (advanced)
• Market Blend Weight: 0-1, how much to blend with market (0 = model only)

Example: Chiefs vs Bills

Understanding QB Names

QB names should match how they appear in NFL data. Common formats:

• "Patrick Mahomes" (not "P. Mahomes" or "Mahomes")
• "Josh Allen" (not "J. Allen")
• "Lamar Jackson" (full name)

Tip: If unsure, check recent game logs or use the QB's full name as it appears in official NFL stats.

Market Calibration (Advanced)

Market calibration blends your model's predictions with betting market consensus:

• w = 0.0: Use only model predictions (default)
• w = 0.5: Equal blend of model and market
• w = 1.0: Use only market predictions

When to use: If you have betting market data and want to incorporate market wisdom. Requires a CSV file with columns: game_id, spread_home, total, home_team, away_team.

Running Backtests: Detailed Guide

What is a Backtest?

A backtest evaluates model accuracy by:

1. Running simulations on historical games
2. Comparing predictions to actual outcomes
3. Calculating metrics (Brier score, log loss, calibration)

When to Run Backtests

• After training a new model (validate performance)
• Comparing different models
• Evaluating model improvements
• Understanding model strengths/weaknesses

Backtest Parameters

• Seasons: Which seasons to test on (e.g., "2023,2024")
• Simulations per Game: Default 5,000 is good
• Model: Which model to test (default: active model)
• Market CSV: Optional, for comparing to market

Example: Testing 2023 Season

Understanding Backtest Results

The backtest report includes:

• Brier Score: Lower is better (measures prediction accuracy)
• Log Loss: Lower is better (measures probability calibration)
• Calibration Curve: Shows if probabilities match actual frequencies
• Game-by-Game Results: Predictions vs actual for each game

Step-by-Step Examples

Example 1: Complete Beginner Workflow

Example 2: Comparing Two Models

Example 3: Using Market Calibration

game_id,spread_home,total,home_team,away_team
2024_1_KC_BUF,-3.5,52.5,KC,BUF
2024_1_SF_DAL,7.0,48.0,SF,DAL

Interpreting Results

Simulation Results

Win Probability

Example: "58.2%" means the home team wins in 58.2% of simulations.

• 50% = toss-up
• >60% = strong favorite
• <40% = strong underdog

Score Distribution

Example: "Home: 27.3 (std: 7.2), Away: 24.1 (std: 6.8)"

• Mean: Average score across all simulations
• Std: Standard deviation (higher = more uncertainty)

Quantiles

Example: "Home Score 5th percentile: 15, 95th percentile: 38"

• 5th percentile: Score exceeded in 95% of simulations (low end)
• 50th percentile (median): Middle score
• 95th percentile: Score exceeded in only 5% of simulations (high end)

Interpretation: "There's a 90% chance the home team scores between 15 and 38 points."

Spread

Example: "Spread: 3.2 (std: 10.1)"

• Mean: Average point difference (home - away)
• Positive = home team favored
• Negative = away team favored

Backtest Results

Brier Score

Measures prediction accuracy. Range: 0 (perfect) to 1 (worst).

• <0.20 = Excellent
• 0.20-0.25 = Good
• >0.25 = Needs improvement

Log Loss

Measures probability calibration. Lower is better.

• <0.50 = Excellent
• 0.50-0.70 = Good
• >0.70 = Needs improvement

Calibration Curve

Shows if predicted probabilities match actual frequencies.

• Diagonal line: Perfect calibration
• Above diagonal: Model is overconfident
• Below diagonal: Model is underconfident

Troubleshooting

Common Issues

Training Job Stuck on "Queued"

Problem: Job never starts running

Solutions:

• Check Celery workers are running: sudo systemctl status gamesim-celery-train
• Restart workers if needed: sudo systemctl restart gamesim-celery-train
• Check logs: sudo journalctl -u gamesim-celery-train -f

Simulation Returns Unexpected Results

Problem: Win probabilities seem wrong

Solutions:

• Verify QB names match NFL data exactly
• Check team abbreviations are correct (3-letter codes)
• Ensure model is trained on recent seasons
• Try increasing number of simulations (10,000+ recommended)

Model Training Fails

Problem: Training job fails with error

Solutions:

• Check job logs for specific error message
• Verify data is available for selected seasons
• Try smaller season range first
• Check disk space: df -h
• Check memory: free -h

Can't Find QB in Model

Problem: QB name not recognized

Solutions:

• Use full name as it appears in NFL stats (e.g., "Patrick Mahomes" not "P. Mahomes")
• Check if QB played in training seasons
• Try "Unknown" if QB is not in training data (model will use average QB effect)

Getting Help

• Check job logs for detailed error messages
• Review model metrics in Model Registry
• Compare with known-good examples
• Check system resources (CPU, memory, disk)

Quick Reference