unable to use table_import_from_s3 with any path

https://forums.aws.amazon.com/thread.jspa?messageID=957689&#957689 - had success with importing when the object is directly within the bucket you might want to try that.

It's apparently a known bug - quoting from:

https://stackoverflow.com/questions/58616005/import-postgres-data-into-rds-using-s3-and-aws-s3

Currently (2020-10-16) at least with RDS Postgres 12.4, importing files from S3 does not work if the file is not in root of S3 bucket.

Importing file myfile.csv works, importing file mydir/subdir/myfile.csv doesn't work. The latter will give these kinds of errors if permissions and everything else is working:

ERROR: HTTP 403. Permission denied. Check bucket or provided credentials as they may no longer be valid.
This is known issue to AWS and according to AWS Support they are working on the issue.

Also it seems that you need to reboot the RDS instance after adding the roles described in documentation. Otherwise the roles do not have any effect. The need for reboot is not mentioned in the documentation.

In general RDS s3Import gives very confusing error messages. For example, importing 0 length file, gives this error:

(500310) Invalid operation: HTTP 416. Check your arguments and try again.;

Edited by: StuAD on Oct 22, 2020 4:27 AM

We have the same problem using AWS Aurora PostgreSQL compatible DB with engine v.11.8.

Some things we tried without success ...

1. URL-encoding characters other than letters and numbers in the file path.
2. Using Unicode U&'002F' instead of the forward slash (/) character.
3. Explicitly adding the "folder" in a VPC Endpoint policy as a "Resource".
   "arn:aws:s3:::<bucket>/<folder>/*"
4. Using PostgreSQL dollar quoting ($$) instead of the single quote (') character around the file path string.
5. Listing the bucket, file path, region as separate arguments to the table_import_from_s3() function.
6. Using aws_commons.create_s3_uri(<bucket>,<file path>,<region>) as a single argument to table_import_from_s3().
7. Placing a copy of the file in a new folder directly under the bucket "root" with a very simple 7-letter name.

We can import data without any problem IF the file is at the "root" of the bucket.

Wow. This is a really bad P1 bug. Any updates AWS?

We're experiencing the same s3 permission denied error after upgrade to 11.8.

Has there been any update on this?
We are also encountering the same issue.

We started a new service that used an aurora postgres db with engine_version 11.8.

We we're unable to do s3import from subfolders regardless how we setup the policies for our database iam role. Importing from root of the bucket worked.

We recreated the database with version 11.7 and now things work as expected. Not ideal if you have a production environment though.

My work around was to copy the file to the root of the bucket and then perform the aws_s3.table_import_from_s3. Later, deleted the file from root.

UPDATE: (2021-01-11) Upgrading to Aurora PostgreSQL-compatible engine v. 11.9 and ensuring that the path presented to aws_s3.table_import_from_s3() does not begin with a forward slash ("/") enabled successful load of data from .csv.gz files in S3 buckets outside the root (e.g. path has multiple forward slashes) to Aurora DB tables.

I updated my Postgres RDS to 11.9 and still am receiving the error: InternalError_: HTTP 403. Permission denied. Check bucket or provided credentials as they may no longer be valid.

I have double and tripled-checked that the IAM role and policy for this have been create properly.
The query I am using is this:

select aws_s3.table_import_from_s3('table1', '', '(format csv)', '<s3-bucket-name>', 'crunchbase/out/cb-20201008.csv', 'us-east-2' );  

I also have tried this version but get the same result:

select aws_s3.table_import_from_s3('table1', '', '(format csv)',   
                                                            aws_commons.create_s3_uri('<s3-bucket-name>','crunchbase/out/cb-20201008.csv', 'us-east-2')  
                                                            );  

I am not using the Aurora RDS, just a basic Postgres RDS database. If you could provide any insight into how to fix this I would greatly appreciate it!

Looks like a number of these symptoms have been addressed in Aurora PostgreSQL 3.3.2:

https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraPostgreSQL.Updates.20180305.html#AuroraPostgreSQL.Updates.20180305.332