Model accuracy dropped in production. How do you find out why?
Model answer
First, confirm the drop is real. Check that the metric is computed correctly and that labels are not delayed or changed. Then find when it started and what changed around that time: a model release, a code change, an upstream data source or a seasonal event.
Check the data pipeline. Look for broken or missing features, schema changes, new null rates, changed units or encodings, and training-serving skew, where features are computed differently online and offline.
Check drift. Compare current inputs with the training data using summary statistics and tests such as PSI or Kolmogorov-Smirnov. Data drift means the input distribution changed. Concept drift means the relationship between inputs and the target changed, so the same inputs now have different answers.
Slice the metric by segment, region, device or time to see where it fails.
Fix. Repair the pipeline, or retrain on recent data, possibly with a rollback to the last good model in the meantime. Add monitoring for input distributions and prediction quality so the next drop is caught early.
Key points
- Verify the metric and labels, then date the drop and correlate it with changes
- Check pipeline breakage and training-serving skew first
- Distinguish data drift from concept drift
- Slice by segment, then fix, retrain or roll back, and add monitoring
Likely follow-ups
- How do you monitor a model when labels arrive weeks later?
- How often should the model be retrained?
Common mistakes
- Retraining immediately without finding the cause
- Looking only at the global metric and missing a failing segment
- Ignoring bugs in feature computation online