User feedback storage and metric visualization system with AWS Grafana and RDS?
Hi!
I need to retrieve from my app user feedback from diferent parts of my application, my idea is to get that feedback, sending to an SQS queue and then execute a lambda to store that feedback inside an RDS Postgresql DB, to then be able to visualize this feedback metrics using Grafana.
Is this a doable approach? or what is the best approach to store user feedback and see and generate metrics based on that data?
- Language
- English
asked 2 years ago266 views
- Newest
- Most votes
- Most comments
Your proposed approach for storing user feedback and visualizing metrics using AWS services is certainly doable and aligns well with common architectural patterns. Let's break down the components and discuss the approach:
-
Collecting feedback: Your idea of sending user feedback to an SQS queue is a good start. This provides a buffer and helps decouple the feedback collection from the processing, allowing for better scalability and reliability.
-
Processing and storage: Using a Lambda function triggered by the SQS queue to process the feedback and store it in an RDS PostgreSQL database is a solid approach. This serverless architecture can handle varying loads efficiently.
-
Visualization with Grafana: Amazon Managed Grafana is an excellent choice for visualizing and analyzing your feedback data. It's a fully managed service that allows you to create interactive dashboards and share them within your organization.
To implement this system:
- Set up an SQS queue to receive feedback from your application.
- Create a Lambda function that processes messages from the SQS queue and inserts data into your RDS PostgreSQL database.
- Configure Amazon Managed Grafana and set up your RDS PostgreSQL as a data source.
- Create dashboards in Grafana to visualize your feedback metrics.
This approach allows you to:
- Scale easily to handle varying amounts of feedback
- Analyze trends and patterns in user feedback over time
- Create custom visualizations and alerts based on your specific needs
Amazon Managed Grafana supports various data sources, including RDS PostgreSQL, making it easy to query and visualize your feedback data. You can create interactive dashboards to monitor key metrics, set up alerts for specific feedback patterns, and share insights across your organization.
If you want to enhance this system further, you could consider:
- Using Amazon CloudWatch in conjunction with Grafana for additional monitoring capabilities
- Implementing Amazon CloudWatch RUM (Real User Monitoring) to capture more detailed user experience data
- Utilizing Amazon QuickSight as an alternative or complement to Grafana for business intelligence and reporting
Overall, your proposed approach is sound and can be effectively implemented using AWS services. It provides a flexible and scalable solution for storing user feedback and generating insightful metrics.
Sources
Amazon Managed Grafana for dashboarding and visualization - AWS Prescriptive Guidance
Real User Monitoring with Amazon CloudWatch RUM and Amazon Managed Grafana | AWS Cloud Operations & Migrations Blog
Amazon Managed Grafana - Introduction to DevOps on AWS
Use your Grafana workspace - Amazon Managed Grafana
answered 2 years ago
Hello,
to add to the answer above, it also depends on the volume of feedback you are collecting, as if this is rather low you could think about bypassing the SQS and Lambda piece and writing directly to the DB from your application, not the most fancy approach but it all depends on the requirements. Also, if the feedback needs to be processed in real time, SQS and Lambda may not be the best solution.
A possible improvement could be to include Amazon Comprehend for natural language processing to extract sentiment, entities, and key phrases from feedback text
answered 2 years ago
Hi! Out of curiosity, what should I use then if SQS and lambda are not the right approach to store the feedback as soon as the user sends it? My idea is to store the feedback when the user sends it, but I dont need to have it in "real time" it can be delayed.
Hi,
for extremely high-volume, real-time data ingestion and processing, you could use Amazon Kinesis Data Streams in combination with Kinesis Data Analytics. This setup can handle millions of records per second and provide real-time analytics.
If you want to stay with the proposed architecture and move toward real-time, instead of using a standard SQS queue, use an SQS FIFO (First-In-First-Out) queue with Lambda. FIFO queues guarantee the order of messages and can be processed nearly instantaneously. On the Lambda side, you can configure your function to process messages in smaller batches with a shorter timeout, and you can also increase concurrency.
Hope that helps.
Yeah! actually that is really helpful, thanks!
I would consider timescaledb if at all possible, because performance looking at time series data in grafana is much better over wide time ranges if you've got the ability to use time_bucket() and time_bucket_gapfill() as well as all the available hyperfunctions designed specifically for dealing with time-series data. I don't know if you can add timescale extensions and the timescaledb toolkit to RDS postgres.
I see, thanks for that point of view too, I will think of it!