Training Large-Scale Transformer Models with DeepSpeed and Habana Gaudi Accelerators on AWS

Training large language models (LLMs) with billions of parameters presents significant challenges, including high memory and computational requirements, as well as extended training times. To address these issues, DeepSpeed has emerged as a reliable solution for optimizing large-scale model training by offering state-of-the-art scaling efficiency and system performance. In this article, we provide a comprehensive guide on how to effectively train large-scale transformer models using DeepSpeed and Habana Gaudi accelerators on Amazon Web Services (AWS).

Setting up training with Habana Gaudi-based Amazon EC2 DL1 instances

Intel’s Habana Gaudi-based Amazon EC2 DL1 instances are specifically designed for training large-scale models, offering high-bandwidth accelerators and low-latency interconnects to enhance performance. The dl1.24xlarge instance, for example, provides 32 Gaudi accelerators, delivering exceptional performance for data and model parallelism workloads.

Scaling results for encoder-type transformer models

DeepSpeed ZeRO (Zero Redundancy Optimizer) stage 1 optimizations can significantly improve scaling efficiency, which is critical for training large transformer models. By leveraging data parallelism, a 1.5-billion-parameter BERT model can be trained efficiently on multiple Habana Gaudi-based DL1 instances. This allows researchers and engineers to achieve faster training times and reduced memory usage, both vital aspects for handling models with massive parameter counts.

Benefits of using the BF16 data type

One significant advantage of integrating Gaudi accelerators is their native support for the BF16 data type. The BF16 data type contributes to improved training performance by reducing memory size and memory-bound computations throughout the training process. As a result, Gaudi accelerators can offer high performance while maintaining a lower memory footprint and reduced training times.

Pre-training model convergence within 16 hours

With DeepSpeed and Gaudi accelerators, training a 1.5-billion-parameter BERT model on the wikicorpus-en dataset can now be achieved within 16 hours, a significant improvement over previous training times. This reduced training time enables faster research iterations, leading to more effective experimentation and model development.

Training setup with AWS Batch

To set up a managed compute cluster for distributed training, AWS Batch proves to be a powerful tool. It simplifies the process of launching large-scale containerized training jobs on AWS, handling resource provisioning, job scheduling, and management efficiently. By integrating AWS Batch with DeepSpeed and Habana Gaudi accelerators, users can further achieve greater control and efficiency in their large-scale transformer model training tasks.

The combination of DeepSpeed, Habana Gaudi accelerators, and AWS provides an optimized solution for training large-scale transformer models. This approach not only enhances scaling efficiency and performance but also enables faster pre-training times and improved research quality. Data scientists, machine learning engineers, and researchers can benefit significantly from this powerful solution, setting the stage for unparalleled advancements in the field of artificial intelligence and natural language processing. So, why wait? Start exploring this approach and transform your model training processes today.