Analytics Store for run time slice and dice

Question

Hello Everyone,

We are trying to build an analytics system over Jira data. Something similar to https://aws.amazon.com/blogs/big-data/ingesting-jira-data-into-amazon-s3/ . We are ingesting data in S3 , then using AWS glue spark job we transform the data and push it into OpenSearch. OpenSearch works well with aggregation, slicing, and dicing. However, it is expensive. To reduce the cost we are planning pre-comute few reports and keep it in S3. However, we cannot achieve slicing and dicing with S3 in run time. We have the below option. Would love to hear your thoughts and please recommend if you think they are other choices in AWS.

1) RDS/Postgress - But we think it will have a scalability issue when data grows

2) OpenSearch - Again it can be expensive

Answer

One option could be [Amazon QuickSight](https://aws.amazon.com/quicksight/), you could load the pre-computed data into is in-memory engine SPICE and create visual to easily slice and dice.

As for a relational Database aimed at OLAP that could easily scale horizontally you could look at [Amazon Redshift](https://aws.amazon.com/redshift/), there is also a preview of a [serverless](https://docs.aws.amazon.com/redshift/latest/mgmt/working-with-serverless.html) deployment option.

more details on how you want to use the data (through visualization? SQL queries?) it could be useful to give a more precise answer.

hope this helps

Analytics Store for run time slice and dice

관련 콘텐츠