Real Time Data Ingestion
IHAC working on enabling real time Data ingestion on their S3 based Datalake with the objective of showing near-realtime data on QuickSight Dashboards. As of now, they are heavily relying on QuickSight SPICE engine for better Dashboard performance since direct Athena queries from QuickSight are not so performant. With Realtime data flowing into the Datalake, they need to refresh QuickSight SPICE datasets very frequently. However, they are hitting a limitation of 32 calls per 24 hours on CreateIngestion API to refresh Quicksight SPICE dataset. This is a serious blocker for them. Could you please suggest how can they overcome this limitation and move forward.. An error occurred (LimitExceededException) when calling the CreateIngestion operation (reached max retries: 2): Rate exceeded for CreateIngestion on this dataset. Please retry after 1304 minutes
- Tópicos
- Estatísticas
- Idioma
- English
feita há 2 anos322 visualizações
- Mais recentes
- Mais votos
- Mais comentários
Hi,
Since you are allowed to 32 full refreshes per day, you are probably using the Enterprise edition of QuickSight.
In that case, you can also use the incremental refresh (i.e. up to every 15 min) in order to refresh your SPICE data. Did you consider it?
Best,
Didier
Thanks for the recommendation Didier. Cx is looking for near real time data refresh. Incremental refresh may run with out limit exceptions but there is a 15 minute delay which is long time for this requirement.