What is the right VPC setup for loading data from S3 to Aurora

0

I have an Aurora DB setup with MYSQL8.0 in Oregon (This is Aurora serverless v2 and there is no Data API in this version). This has VPC, subnets, VPC security group and public access turned on. The main route table of the VPC has a route with com.amazonaws.us-west-2.s3 as destination and an endpoint of the vpc as target. When I try loading data into aurora from S3 (the bucket is in N.California) using the LOAD DATA FROM S3 's3-us-west-1:.... statement, I keep getting ERROR 1815 (HY000): Internal error: Unable to initialize S3Stream. I have also rebooted the DB's writer instance after all the IAM and VPC changes. What am I missing?

  • Hi,

    Can you confirm what your IAM policy allows for this access? I would check both the S3 bucket policy and the role associated with your Aurora setup.

  • Thanks for your comment. The role associated with Aurora seems to have the right access. What would be the right S3 bucket policy?

  • { "Version": "2012-10-17", "Id": "s3accesspolicy", "Statement": [ { "Sid": "VPC endpoint access to S3", "Principal": "", "Action": "s3:GetObject", "Effect": "Allow", "Resource": ["arn:aws:s3:::bucket","arn:aws:s3:::bucket/"], "Condition": { "StringEquals": { "aws:sourceVpce": "vpce-ID" } } } ] }

    I am using a policy similar to this for the buckets, but still get the S3Stream error for buckets both within and outside the DB's region

1 Answer
1

You are loading data from another region, so the VPC gateway endpoint will not be used, the route table has a prefix list of IPs for S3 in Oregon. Your VPC needs to have an Internet Gateway and the route table associated with the Aurora subnets needs a '0.0.0.0/0' route to the IGW. See: Gateway endpoints .

Traffic that's destined for the service (Amazon S3 or DynamoDB) in a different Region goes to the internet gateway because prefix lists are specific to a Region.
profile pictureAWS
EXPERT
kentrad
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions