Skip to content

Cross-regional S3 access from Lambda

0

I have a lambda that needs to be in a us-east-1 VPC to access some resources, but also needs to write to an S3 bucket which needs to be in a different region. (I must use the python boto3 client for this.)

This is not going well. I'm getting timeouts no matter what I try, so this is a routing/security group issue. Yes, the lambda is in a private subnet.

The usual S3 vpce doesn't do cross-regional access. (Everything works FINE if I'm access a bucket in us-east-1 with it, however). Trying the "config=boto_config.Config(s3={'addressing_style':'path'}" config option to boto3.client does not help. Setting a global S3 interface endpoint in the private subnets does not work.

Help?

2 Answers
3

Hello.

Since VPC endpoints can only access S3 in the same region, I think you will need to create a NAT Gateway etc. to be able to access S3.
Additionally, as introduced in the document below, it is also possible to perform VPC peering with a VPC in another region and route it through the interface VPC endpoint.
https://repost.aws/knowledge-center/vpc-endpoints-cross-region-aws-services

Or you should be able to access it by setting up a multi-region access point like you have set up.
When using Multi-Region Access Points, you need to either configure private DNS or specify in your Lambda code to access "accesspoint.vpce-.accesspoint.s3-global.us-east-1.vpce.amazonaws.com" as the VPC endpoint.
Also, since a timeout error is occurring, please check whether HTTPS is allowed in the inbound rules of the VPC endpoint's security group.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/MultiRegionAccessPointsPrivateLink.html

If private DNS is not enabled on the VPC endpoint, you can access it by setting "endpoint_url" as shown below.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/privatelink-interface-endpoints.html#accessing-bucket-and-aps-from-interface-endpoints

client = boto3.client('s3', endpoint_url='https://accesspoint.vpce-1a2b3c4d-5e6f.s3.us-east-1.vpce.amazonaws.com')
EXPERT

answered 2 years ago

EXPERT

reviewed 2 years ago

  • I'm getting an immediate "cannot connect to endpoint URL" error. Note that the operation is "client.put_object", so I also tried 'https://bucket.vpc-...', with the same effect.

1

S3 supports IPv6 on its public endpoints, so instead of setting up a NAT gateway and paying both hourly and traffic-based fees for it, or setting up VPC interface endpoints that also involve both hourly and per-gigabyte fees, you could add IPv6 addresses to your VPC and the subnets used by the Lambda function to make it a dual-stack VPC, attach an egress-only internet gateway to it, and route the IPv6 default route ::0/0 towards the new egress-only IGW. Also permit outbound IPv6 traffic to ::0/0 on the TCP port 443 in the Lambda function's security group.

You can use access policies to prevent the Lambda execution role from accessing S3 unless the request arrives either through the VPC endpoint in the local region or the IPv6 CIDR for cross-region requests.

You'll still have to pay cross-region traffic fees, but this setup would avoid hourly and per-gigabyte fees for the NAT gateways, because egress-only IGWs are free, the same as regular IGWs, and you could still restrict access based on network origin by using the aws:SourceVpc or aws:SourceVpce condition keys for S3 access within the region and aws:SourceIp for cross-region S3 access.

You can attach a policy statement like this one to the Lambda function's execution role to prevent it from accessing any AWS resource from outside the IPv6 CIDR of the VPC or any of the VPC endpoints in the VPC. Replace vpc-00000000000000000 with your VPC ID and fd6f:59a7:101c::/56 (a dummy address) with the IPv6 CIDR:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyAllAccessExceptFromPermittedNetworkOrigins",
      "Effect": "Deny",
      "Action": "*",
      "Resource": "*",
      "Condition": {
        "StringNotEqualsIfExists": {
          "aws:SourceVpc": ["vpc-00000000000000000"]
        },
        "NotIpAddressIfExists": {
          "aws:SourceIp": ["fd6f:59a7:101c::/56"]
        },
        "BoolIfExists": {
          "aws:ViaAWSService": "false"
        }
      }
    }
  ]
}
EXPERT

answered 2 years ago

EXPERT

reviewed 2 years ago

  • I've added an eigw to the route table for the private subnets, and added ::/0 access on 443 to the lambda security group. I'm continuing to get connection timeouts, although the endpoint is now s3.amazonaws.com regardless of the region I set in the client. I have also tried setting config=botocore.config.Config(s3={'addressing_style':'path'}), with the same results.

    I've not torn down the global s3 endpoint on the subnets, but, given the other answer, it seems unlikely that they are interfering. Likewise, my console shows the private subnets routed properly to their route table. The route table has entries for the vpc, (ipv4 & ipv6), the nat for ipv4, and the eigw. Removing the nat from the route table, and I still get a timeout, although it routes to the regional endpoint if use the target region in the client initialization (but not if I use the lambda's region).

  • I suggest you try using the normal virtual hosting style endpoints, and specifying the bucket's region when instantiating the S3 client, such as like this: s3 = boto3.client("s3", region_name="eu-west-1"). This will avoid an extra roundtrip first asking S3 which region hosts the bucket. To be sure, did you assign IPv6 CIDRs also to the subnets where the Lambda function is connected and not just the VPC?

  • Yes, my subnets have ipv6 CIDRs associated. Using this exact config still gets the timeout.

  • To be fair, I've been directed to just write the data to a same-region bucket. We'll revisit this later.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.