How to saturate network through with multiple NICs for s3 downloading?

Hi, I'm trying to get s3 downloading fast. Right now, I have a p4d, and I have setup the NICs with 4 cards. The s3 client, somehow only use the primary NIC, so it never goes beyond the 100Gbps. I check the documentation for s3 crt client, and it seems no place for the local interface bindings.

While in the meantime, I also allocate a m6in.32xlarge, which claim to have 200Gbps network bandwidth. For m6in.32xlarge, it says,

Each network interface attached to a network card can achieve a maximum of 170 Gbps

And with this I can go beyond 100Gbps.

I'm wondering how can s3 client be configured to use multiple NICs to fully utilize the throughput? I checked the cpp sdk and aws cli, none of them state this clearly.

주제

스토리지 컴퓨팅 네트워킹 및 콘텐츠 전송

태그

아마존 심플 스토리지 서비스 고성능 컴퓨팅 네트워킹 및 콘텐츠 전송

언어

English

fish6one

질문됨 9달 전259회 조회

1개 답변

최신
최다 투표
가장 많은 댓글

이 답변이 도움이 되었나요?커뮤니티가 여러분의 지식을 활용할 수 있도록 정답을 찬성하세요.

Hi,

assuming that you use S3 sync (or cp) to download, there are some parameters that you can change at this level (depending on your requirements and data structure) to improve throughput and to better use the available h/w: see https://repost.aws/knowledge-center/s3-improve-transfer-sync-command

In particular, the section "Modifying the AWS CLI configuration value for max_concurrent_requests" is relevant even if you don't copy from S3 to S3.

Also, a deep read of https://repost.aws/knowledge-center/s3-optimize-transfer will help

Best,

Didier

전문가

Didier_Durand

답변함 9달 전

iBehr 전문가
9달 전
Adding additional NICs will not get you additional bandwidth on EC2. Each instance is rated for the max bandwidth through the one default NIC. Adding NICs in additional subnets gets to more IPs/interfaces for things like firewalling/routing.
fish6one
9달 전
I think it's not true according to the doc of aws m6in.32xlarge:

*For 32xlarge and metal sizes, at least 2 elastic network interfaces, with each attached to a different network card, are required on the instance to achieve 200 Gbps throughput. Each network interface attached to a network card can achieve a maximum of 170 Gbps. For more information, see Network cards.

The for p4d, they have 4 network cards actually. Each NIC can be added to different device. So does the magic happened inside hardware layer? Like one NIC can fully utilize 4 physical network cards?

How to saturate network through with multiple NICs for s3 downloading?

관련 콘텐츠