Open Source Summit India 2026

PyTorch is a widely adopted library for deploying ML workloads. It provides robust support for resource optimization, including CPU offload capabilities, memory and distributed workload management. The primary objective of any ML workload is to achieve maximum performance during inference or training. PyTorch enables acceleration of these workloads through support for torch.compile and distributed library capabilities. This session will cover the internals of the torch.compile stack, including the reasons for performance improvements such as internal graph representation optimization. These benefits will be demonstrated in the hands-on part of the session. While torch.compile can provide performance improvements on a single GPU, most production workloads require multiple GPUs to significantly reduce overall execution time. This session will examine how tensor distribution across GPUs is performed using various parallelization techniques, including data, tensor, and pipeline parallelism. The hands-on part of the session will involve implementing these parallelization techniques within a mini-PyTorch implementation, enabling practical understanding of distributed training strategies.