Automating data updates on Quicksight



I have multiple different CSV files that I receive via email, on a day to day basis.

I would like to build a pipeline to ensure that any datasets that I receive through emails, can be picked up by Quicksight automatically and all the updates are done without much intervention.

As a result I thought of the following:

  1. Update all the CSVs (from email) I get each day, into S3 Buckets.

  2. Point the S3 buckets to AWS Athena.

  3. Quicksight will use Athena to do automatic visual updates.

For this to work, I will have to ensure that the data manipulation is done outside AWS, and then manually update CSVs inside S3 buckets on my own, each day.

Another alternative idea is the following:

  1. Place all data into S3 Buckets.

  2. Use Glue data brew to manipulate data.

  3. Store it back to S3 buckets.

  4. Use Quicksight to visualise data (pointing it back to S3 buckets).

Can someone please advise of a better method than this ? I am looking for more automation and easier methods of manipulating data for Quicksight to update each day. My data columns don't change, but new date fields and values are added each day.

asked 2 years ago528 views
1 Answer

Hi, @Dinesh

I understand that you want to automate the following three things.

  1. Importing data into the lake

  2. Processing of intermediate data

  3. Drawing

  4. is not mentioned in detail, so ignore it.

  5. can be realized by using the automatic update function of SPICE.
    Before doing 1), you need to sort out where all your data comes from and how you can get it. In many cases it can be automated with Glue or Lambda / Step Functions.

profile picture
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions