November 28, 2024
Xunyi Zhao will defend his PhD thesis, titled Optimizing Memory Usage when Training Deep Neural Networks on Tuesday 10th of December at 10:30 AM in the Ada Lovelace room at INRIA. The presentation will be in English.
Abstract: Artificial Intelligence (AI) has seen remarkable growth in recent years, proving its utility across a wide range of fields including image recognition, natural language processing and autonomous systems. This success is largely driven by access to increasingly large datasets and the development of Deep Neural Networks (DNNs) with greater complexity and size, allowing AI systems to achieve unprecedented levels of performance. However, the growing scale of tasks presents significant challenges, particularly when it comes to training these massive models on devices with limited memory capacity. Efficiently managing memory during training has become a critical focus to ensure that even resource-constrained systems can handle complex AI tasks. There are several strategies to address memory constraints in neural network training. Memory footprints can be distributed across multiple devices or compressed using specialized algorithms that minimize information loss. This thesis focuses on lossless memory-saving techniques, primarily applied to single-device scenarios. The key approaches include reducing the memory cost of intermediate activations by discarding and recomputing them when required, and managing parameter memory costs by swapping them to larger capacity RAM. We have developed optimization algorithms that integrate these techniques, effectively lowering peak memory usage while maintaining efficient training iteration times. Our solutions are implemented in an open-source framework, tested and compatible with leading AI libraries such as PyTorch, HuggingFace, and DeepSpeed.