Heidelberg AI News

Official announcement · AWS Machine Learning ·

Detecting and recovering from silent data corruption on Amazon SageMaker HyperPod

Silent data corruption is a GPU fault that returns wrong but plausible numbers with no error, quietly poisoning distributed training. This post adds three checks to a PyTorch DDP loop to detect it, localize the faulty node on Amazon SageMaker HyperPod, and hand the node to automatic recovery for replacement.

Read at aws.amazon.com The opening of the article at aws.amazon.com.

In this story

A tool or model opens in the Heidelberg AI Curriculum: what it does, what it costs, how to start. A repo opens on GitHub.

Related stories

All stories

OfficialHugging Face

Transformers Model Integration Contracts

On July 30, 2026, Transformers enabled PyTorch's scaled dot-product attention (SDPA) for T5. As a side effect, UMT5's decoder could attend to future tokens. During training, it could peek at the answer it was supposed to predict. Nothing crashed, and the bug stayed in main for almost two months.

VideoIBM Technology

What Is torch.compile? Dynamo, PyTorch & Faster AI Models Explained

The video explains how PyTorch’s torch.compile optimizes models through Dynamo, AOT Autograd, and Inductor, including lazy compilation, graph capture, kernel fusion, and hardware-specific code generation. It also shows why speedups vary and how graph breaks, recompilation, and incorrect benchmarking can limit performance.