Quicksight Dataset suddenly cant refresh from S3 anymore

Question

I have a Quicksight dataset that’s been working fine for months pulling data from S3 via a manifest file but all of a sudden all the refreshes started failing with the errors below since yesterday:
FAILURE_TO_PROCESS_JSON_FILE=
Error details:
S3 Iterator: S3 problem reading data

I’ve double checked the manifest file format and the S3 bucket permissions for Quicksight and everything seems fine and nothing has changed on our end for this to suddenly stop working out of the blue…

Manifest file:

```
{
"fileLocations": [
{
"URIPrefixes": [
"https://s3.amazonaws.com/solar-dash-live/"
]
}
],
"globalUploadSettings": {
"format": "JSON"
}
}
```

The error I get in the email alert is different and says "Amazon QuickSight couldn't parse a manifest file as valid JSON." However I verified that the above JSON is formatted correctly.

Also, if I create a new dataset with the same manifest file, it will show the data in the preview tool but its just the Refresh that fails so clearly the manifest file is formatted correctly if Quicksight is able to initially pull data from S3 but only fails later.

Answer

Hello,
There are several possible reasons for the failed refreshes.

1.	The format of the data going into the S3 bucket has changed. I understand that you have verified the format,  however please recheck it with the previous format.
2.	The application generating the data is having a problem and is uploading corrupted data.

3.	Something is causing the manifest file to change after the initial dataset creation but before the refresh.

4.	Without access to the bucket policy itself, it is difficult to confirm whether the permissions are indeed correct.

The process that occurs internally to handle a refresh also depends on the way QuickSight connects to the data store. Unfortunately, we will need more information to narrow down the issue.

For more information, please see the documentation on refreshing data in QuickSight:
https://docs.aws.amazon.com/quicksight/latest/user/refreshing-data.html

If you are using a SPICE dataset, this documentation may help you ensure that the refresh is set up properly:
https://docs.aws.amazon.com/quicksight/latest/user/refreshing-imported-data.html

Quicksight Dataset suddenly cant refresh from S3 anymore

Relevant content