使用AWS re:Post即您表示您同意 AWS re:Post 使用条款
AWS re:Postre:Post

How to saturate network through with multiple NICs for s3 downloading?

0

Hi, I'm trying to get s3 downloading fast. Right now, I have a p4d, and I have setup the NICs with 4 cards. The s3 client, somehow only use the primary NIC, so it never goes beyond the 100Gbps. I check the documentation for s3 crt client, and it seems no place for the local interface bindings.

While in the meantime, I also allocate a m6in.32xlarge, which claim to have 200Gbps network bandwidth. For m6in.32xlarge, it says,

Each network interface attached to a network card can achieve a maximum of 170 Gbps

And with this I can go beyond 100Gbps.

I'm wondering how can s3 client be configured to use multiple NICs to fully utilize the throughput? I checked the cpp sdk and aws cli, none of them state this clearly.

主题
存储计算网络和内容交付
标签
亚马逊简单存储服务 (S3)高性能计算网络和内容交付
语言
English
fish6one
已提问 9 个月前258 查看次数
1 回答
0

Hi,

assuming that you use S3 sync (or cp) to download, there are some parameters that you can change at this level (depending on your requirements and data structure) to improve throughput and to better use the available h/w: see https://repost.aws/knowledge-center/s3-improve-transfer-sync-command

In particular, the section "Modifying the AWS CLI configuration value for max_concurrent_requests" is relevant even if you don't copy from S3 to S3.

Also, a deep read of https://repost.aws/knowledge-center/s3-optimize-transfer will help

Best,

Didier

profile pictureAWS
专家
Didier_Durand
已回答 9 个月前
  • iBehr 专家
    9 个月前

    Adding additional NICs will not get you additional bandwidth on EC2. Each instance is rated for the max bandwidth through the one default NIC. Adding NICs in additional subnets gets to more IPs/interfaces for things like firewalling/routing.

  • fish6one
    9 个月前

    I think it's not true according to the doc of aws m6in.32xlarge:

    *For 32xlarge and metal sizes, at least 2 elastic network interfaces, with each attached to a different network card, are required on the instance to achieve 200 Gbps throughput. Each network interface attached to a network card can achieve a maximum of 170 Gbps. For more information, see Network cards.

    The for p4d, they have 4 network cards actually. Each NIC can be added to different device. So does the magic happened inside hardware layer? Like one NIC can fully utilize 4 physical network cards?

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈，并能促进提问者的职业发展。

回答问题的准则

相关内容