Quicksight S3 Manifest Capacity Solution

Question

We store data in S3 hourly so that means 24 files per day * 30 days a month = 720 files a month but Quicksight has a 1000 file limit per manifest file when importing data from s3. What is the best practice here? Should I create a new manifest file each month and then combine the datasets into one?

Here is a link to the documentation: https://docs.aws.amazon.com/quicksight/latest/user/data-source-limits.html

Answer

How about using Amazon Athena?
 https://docs.aws.amazon.com/athena/latest/ug/what-is.html

Amazon Athena can refer to multiple files in S3 like a database.
If you read the data via Amazon Athena, you should be able to import it into QuickSight without any problems.

Answer

Hello ,

Yes you would be correct in your understanding of how the limit works.

I can confirm that as soon as you reach the limit of 1000 files within any type of S3 [manifest file format](https://docs.aws.amazon.com/quicksight/latest/user/supported-manifest-file-format.html) you would be met with the following message:

`
The S3 data source exceeds QuickSight’s limits. Please see data source limits for more information`

Therefore the best workaround would be to use [Amazon Athena](https://docs.aws.amazon.com/athena/latest/ug/what-is.html) , as this would allow you to have as many files in your S3 bucket as you may need  and thereafter you will still be able to upload your data to QuickSight via Athena.

Quicksight S3 Manifest Capacity Solution

相关内容