Video · IBM Technology ·
AI Model Debugging: Find the First Divergence Across AI Accelerators
In short
The video explains how to debug AI models that produce different results across accelerators by comparing intermediate tensors between a validated reference execution and the target system. Finding the first observed divergence narrows a model-wide failure to a smaller region that can be isolated, fixed, and protected with a regression test.
Key points
- Trace potential failures across the model, compiler, runtime, and accelerator layers.
- Distinguish expected floating-point variation from correctness bugs using operation-appropriate tolerances.
- Compare tensor shapes, data types, maximum absolute differences, relative error, and cosine similarity.
- Find the first observed divergence between the last matching checkpoint and first failing checkpoint.
- Use minimal tests to isolate operations such as normalization and inspect precision, layout, broadcasting, and lowering.
- Rerun the full model and preserve the reproducer as a CI/CD regression test.
Written by an AI model from the video’s captions and description.
The same AI model can produce different results across accelerators. Kaoutar El Maghraoui explains AI model debugging by finding the first observed divergence between executions. Learn how tensor comparisons, tolerances, and minimal tests help isolate failures.



