Training LLM on p5en.48xlarge (H200×8) is 2.5× Slower than on My Personal Server (H100×8)

Question

I am currently training an LLM on an AWS p5en.48xlarge instance (8× H200 GPUs). Using the exact same code, data, and model, I found that training is about 2.5 times slower compared to my personal server with 8× H100 GPUs.

While I did not necessarily expect the H200s to outperform the H100s, the fact that training is significantly slower on the p5en.48xlarge seems unusual. Could there be something I am missing in terms of setup or configuration?

My current instance settings are as follows:
Instance: p5en.48xlarge (8× H200)
AMI: Deep Learning Base AMI with Single CUDA (Ubuntu 22.04)

re:Post Agent · Answer

The performance difference you're experiencing between your personal H100 server and the p5en.48xlarge instance is unexpected, as the H200 GPUs should theoretically offer similar or better performance than H100s for most workloads.

Several factors could be contributing to this performance discrepancy:

1. **NVLink/NVSwitch Configuration**: The p5en.48xlarge instances have 900 GB/s NVSwitch for GPU-to-GPU communication. Check if your personal server has a different NVSwitch configuration that might be more optimized for your specific workload.

2. **Software Stack**: The Deep Learning Base AMI might not be optimized for the H200 GPUs. Consider using the latest AWS Deep Learning AMIs (DLAMI) specifically designed to support EC2 P5en instances, which may include optimized drivers and libraries.

3. **CUDA Version and Framework Optimization**: Ensure you're using CUDA versions and ML frameworks that are optimized for H200 GPUs. The H200 is newer hardware and might require specific software optimizations.

4. **Storage Configuration**: The p5en.48xlarge offers 8 x 3.84 TB NVMe SSDs. If your training involves significant I/O operations, check if your storage configuration is optimized. The instance also provides up to 100 Gbps EBS bandwidth, which should be leveraged appropriately.

5. **Network Configuration**: If your training involves distributed operations, verify that you're taking advantage of the EFAv3 networking capabilities (3200 Gbps) of the p5en instance.

6. **Instance Utilization**: Check if there are other processes consuming resources on your instance.

7. **Memory Bandwidth Usage**: While H200s have more GPU memory (141 GB per GPU vs 80 GB for H100), your workload might not be leveraging this additional memory effectively.

I recommend profiling your training job using NVIDIA's tools to identify bottlenecks, and consider consulting with AWS support to ensure your instance is properly configured for optimal performance with your specific LLM training workload.  
**Sources**  
[Specifications for Amazon EC2 accelerated computing instances - Amazon EC2](https://docs.aws.amazon.com/ec2/latest/instancetypes/ac.html)  
[New Amazon EC2 P5en instances with NVIDIA H200 Tensor Core GPUs and EFAv3 networking | AWS News Blog](https://aws.amazon.com/blogs/aws/new-amazon-ec2-p5en-instances-with-nvidia-h200-tensor-core-gpus-and-efav3-networking/)

Training LLM on p5en.48xlarge (H200×8) is 2.5× Slower than on My Personal Server (H100×8)

Add your answer

Relevant content