Durch die Nutzung von AWS re:Post stimmt du den AWS re:Post Nutzungsbedingungen
AWS re:Postre:Post

How to saturate network through with multiple NICs for s3 downloading?

0

Hi, I'm trying to get s3 downloading fast. Right now, I have a p4d, and I have setup the NICs with 4 cards. The s3 client, somehow only use the primary NIC, so it never goes beyond the 100Gbps. I check the documentation for s3 crt client, and it seems no place for the local interface bindings.

While in the meantime, I also allocate a m6in.32xlarge, which claim to have 200Gbps network bandwidth. For m6in.32xlarge, it says,

Each network interface attached to a network card can achieve a maximum of 170 Gbps

And with this I can go beyond 100Gbps.

I'm wondering how can s3 client be configured to use multiple NICs to fully utilize the throughput? I checked the cpp sdk and aws cli, none of them state this clearly.

Themen
SpeicherKalkulationVernetzung & Bereitstellung von Inhalten
Tags
Amazon Simple Storage ServiceHohe RechenleistungVernetzung & Bereitstellung von Inhalten
Sprache
English
fish6one
gefragt vor 9 Monaten258 Aufrufe
1 Antwort
0

Hi,

assuming that you use S3 sync (or cp) to download, there are some parameters that you can change at this level (depending on your requirements and data structure) to improve throughput and to better use the available h/w: see https://repost.aws/knowledge-center/s3-improve-transfer-sync-command

In particular, the section "Modifying the AWS CLI configuration value for max_concurrent_requests" is relevant even if you don't copy from S3 to S3.

Also, a deep read of https://repost.aws/knowledge-center/s3-optimize-transfer will help

Best,

Didier

profile pictureAWS
EXPERTE
Didier_Durand
beantwortet vor 9 Monaten
  • iBehr EXPERTE
    vor 9 Monaten

    Adding additional NICs will not get you additional bandwidth on EC2. Each instance is rated for the max bandwidth through the one default NIC. Adding NICs in additional subnets gets to more IPs/interfaces for things like firewalling/routing.

  • fish6one
    vor 9 Monaten

    I think it's not true according to the doc of aws m6in.32xlarge:

    *For 32xlarge and metal sizes, at least 2 elastic network interfaces, with each attached to a different network card, are required on the instance to achieve 200 Gbps throughput. Each network interface attached to a network card can achieve a maximum of 170 Gbps. For more information, see Network cards.

    The for p4d, they have 4 network cards actually. Each NIC can be added to different device. So does the magic happened inside hardware layer? Like one NIC can fully utilize 4 physical network cards?

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen

Relevanter Inhalt