Quicksight S3 Manifest Capacity Solution

0

We store data in S3 hourly so that means 24 files per day * 30 days a month = 720 files a month but Quicksight has a 1000 file limit per manifest file when importing data from s3. What is the best practice here? Should I create a new manifest file each month and then combine the datasets into one?

Here is a link to the documentation: https://docs.aws.amazon.com/quicksight/latest/user/data-source-limits.html

asked a year ago333 views
2 Answers
0

Hello ,

Yes you would be correct in your understanding of how the limit works.

I can confirm that as soon as you reach the limit of 1000 files within any type of S3 manifest file format you would be met with the following message:

The S3 data source exceeds QuickSight’s limits. Please see data source limits for more information

Therefore the best workaround would be to use Amazon Athena , as this would allow you to have as many files in your S3 bucket as you may need and thereafter you will still be able to upload your data to QuickSight via Athena.

AWS
SUPPORT ENGINEER
answered a year ago
  • Thank you but doesn't that mean I have to pay more money to store the data twice just so I can use quicksight?

0

How about using Amazon Athena? https://docs.aws.amazon.com/athena/latest/ug/what-is.html

Amazon Athena can refer to multiple files in S3 like a database. If you read the data via Amazon Athena, you should be able to import it into QuickSight without any problems.

profile picture
ARBR
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions