What is the right VPC setup for loading data from S3 to Aurora

0

I have an Aurora DB setup with MYSQL8.0 in Oregon (This is Aurora serverless v2 and there is no Data API in this version). This has VPC, subnets, VPC security group and public access turned on. The main route table of the VPC has a route with com.amazonaws.us-west-2.s3 as destination and an endpoint of the vpc as target. When I try loading data into aurora from S3 (the bucket is in N.California) using the LOAD DATA FROM S3 's3-us-west-1:.... statement, I keep getting ERROR 1815 (HY000): Internal error: Unable to initialize S3Stream. I have also rebooted the DB's writer instance after all the IAM and VPC changes. What am I missing?

  • Hi,

    Can you confirm what your IAM policy allows for this access? I would check both the S3 bucket policy and the role associated with your Aurora setup.

  • Thanks for your comment. The role associated with Aurora seems to have the right access. What would be the right S3 bucket policy?

  • { "Version": "2012-10-17", "Id": "s3accesspolicy", "Statement": [ { "Sid": "VPC endpoint access to S3", "Principal": "", "Action": "s3:GetObject", "Effect": "Allow", "Resource": ["arn:aws:s3:::bucket","arn:aws:s3:::bucket/"], "Condition": { "StringEquals": { "aws:sourceVpce": "vpce-ID" } } } ] }

    I am using a policy similar to this for the buckets, but still get the S3Stream error for buckets both within and outside the DB's region

1 個回答
1

You are loading data from another region, so the VPC gateway endpoint will not be used, the route table has a prefix list of IPs for S3 in Oregon. Your VPC needs to have an Internet Gateway and the route table associated with the Aurora subnets needs a '0.0.0.0/0' route to the IGW. See: Gateway endpoints .

Traffic that's destined for the service (Amazon S3 or DynamoDB) in a different Region goes to the internet gateway because prefix lists are specific to a Region.
profile pictureAWS
專家
kentrad
已回答 1 年前

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南