Can I rely on shared subnets for a infrastructure design with thousands of AWS accounts?

0

I'm currently deliberating over an infrastructure design for my AWS setup and would greatly appreciate some insights and advice from the community. The proposed design revolves around the utilization of AWS Organizations, with a focal central account housing essential centralized services, while numerous customer accounts are nested within the organizational structure. The intention is to facilitate seamless communication between the customer accounts and the central services by sharing a subnet across the organization.

In my research, I came across a notable limitation regarding the number of participating accounts in a shared subnet, as outlined in the AWS documentation https://docs.aws.amazon.com/vpc/latest/userguide/amazon-vpc-limits.html#vpc-share-limits. While it's reassuring to know that this limitation can be mitigated through increasing the threshold, I'm keen to understand its scalability, particularly concerning larger organizational scales. For instance, envisioning a scenario with over 10,000 accounts within the organization, I'm curious about the feasibility of proportionally scaling the "Participant accounts per VPC" limit to accommodate such expansion while ensuring the subnet remains efficiently functional.

I'm eager to hear from those with experience or insights into similar infrastructure designs or AWS scalability considerations. Any firsthand experiences, recommendations, or alternative approaches would be immensely valuable in shaping my decision-making process.

Thank you!

  • Can you explain a little more on the services that you wish to expose in the central shared subnet?

  • The central service is comprised of multiple compute instances. The central service would provide compute instances in the customer account with an address to where they can create a TCP connection to send and receive messages to central service.

3 Answers
1

This is a very broad question that you are asking here - as there are so many facets to consider, so I am answering broadly.

I am unsure what types of shared services you had in mind - so cannot comment definitively, however, the purpose of a shared VPC/Subnet is to allow multiple AWS accounts to create their application resources, such as Amazon EC2 instances, Amazon Relational Database Service (RDS) databases, Amazon Redshift clusters, and AWS Lambda functions, into shared, centrally-managed virtual private clouds (VPCs).

So in your scenario - that would mean up to 10,000 different AWS accounts working in a single shared VPC.

This would not be a best practice - and dramatically increases blast radius if something happens such as a compromise to one of the resources that belongs to one account - and that compromise then affects other resources.

If your goal is to provide centralized services that can be consumed by your member accounts - then you should consider using AWS Private Link to vend services out to consumers within each of your many accounts in your organization.

For intra-account network connectivity - consider looking at transit gateway - or even CloudWAN.

AWS
EXPERT
answered 7 months ago
profile picture
EXPERT
reviewed 7 months ago
  • ...the purpose of a shared VPC/Subnet is to allow multiple AWS accounts to create their application resources, such as Amazon EC2 instances, Amazon Relational Database Service (RDS) databases, Amazon Redshift clusters, and AWS Lambda functions, into shared, centrally-managed virtual private clouds (VPCs).

    Thank you for the response. I was hoping that subnets could be shared across VPC boundaries so that the customer accounts could have their own VPCs and subnets, with an additional shared subnet added to their own. However, it sounds like they would simply be allowed to place resources in the central VPC.

1

Some key considerations to get this scale of 10k accounts:

  • Don't rely on traditional CIDR network scheme to enforce routing everything.
  • Think of each VPC and the workload inside this VPC as a microcell that is autonomous and can operate by itself without direct dependency of other VPCs
  • Use VPC endpoints if you need network isolation for this particular workload and need to access to AWS services or other services that your systems provide
  • Design your AWS accounts and VPCs as constructs of isolation. AWS account is your first level of domain isolation. It can be by environment, line of business or department.
  • Shared services are inevitable. But using VPC endpoints or connecting them through Transit Gateways are the best path. If the network overlaps, use VPC endpoint, if not, use Transit Gateway.

But when you think about 10k accounts, it is only required when you are building a multinational or large-scale organization where:

  • Isolation is more important than connecting everyone and everything.
  • Data can still be shared through event bridge, S3 and other services. Networking is less relevant in modern systems.
  • Probably you need to think more about ingress and egress data patterns than making all VPC's be able to talk each other.
profile picture
answered 7 months ago
profile picture
EXPERT
reviewed 7 months ago
1

Hi,

You don't want to share a single subnet across so many accounts: it will bring you lots of management and security headaches.

I would strongly suggest to read this excellent wp: https://docs.aws.amazon.com/pdfs/whitepapers/latest/building-scalable-secure-multi-vpc-network-infrastructure/building-scalable-secure-multi-vpc-network-infrastructure.pdf#vpc-to-vpc-connectivity

The architecture with Transit Gateway seems more appropriate to your use case: see https://docs.aws.amazon.com/whitepapers/latest/building-scalable-secure-multi-vpc-network-infrastructure/transit-gateway.html for pros and cons.

Best,

Didier

profile pictureAWS
EXPERT
answered 7 months ago
profile picture
EXPERT
reviewed 7 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions