Service suggestion for data analysis with outlier and pattern detection

Hello, Since your request involves multiple AWS services and use cases, I recommend working directly with your AWS account team for comprehensive guidance

Thankyou!

Based on your requirements for analyzing employee sick leave data, identifying patterns, and detecting outliers, I would recommend considering Amazon QuickSight with its ML-powered anomaly detection capabilities. While you mentioned concerns about QuickSight's pricing for embedded reports, it's worth noting that you can use QuickSight's ML insights without necessarily embedding reports.

QuickSight's ML-powered anomaly detection can help you identify outliers and patterns in your data without requiring you to specify all potential patterns upfront. It can automatically detect anomalies across millions of data points and provide insights into contributing factors. This aligns well with your need to identify patterns such as disproportionate sick days on Fridays, correlations with long shifts, seasonal trends, and departmental patterns.

For your dataset of 100k-500k records, QuickSight can process this volume of data and provide valuable insights. You can set up ML insights to detect outliers and key drivers in your dataset. The service uses machine learning algorithms to identify anomalies based on historical data, which could help with your requirements like identifying users who take more sick leave on specific days or during certain seasons.

While QuickSight may not directly provide predictive analysis like forecasting who is likely to take sick leave tomorrow, it can show trends and patterns that could inform such predictions.

If you need more advanced predictive capabilities or want to avoid potential session-based pricing issues with QuickSight, you might consider using Amazon SageMaker. SageMaker provides a broad set of machine learning tools that could be used to build custom models for both pattern detection and predictive analysis. However, as you noted, this would require more expertise and development effort compared to a more out-of-the-box solution like QuickSight.

Another option to consider is Amazon OpenSearch (formerly Elasticsearch). It offers real-time anomaly detection capabilities for both streaming and historical data. You could index your sick leave data in OpenSearch and use its anomaly detection features to identify unusual patterns.

Lastly, if you're comfortable with SQL and want to keep things simple, you could use Amazon Redshift ML. This allows you to create, train, and apply machine learning models using SQL commands directly in your Redshift data warehouse. This could be a good middle ground if you want more control than QuickSight offers but don't want to delve into the complexities of SageMaker.

Each of these options has its trade-offs in terms of ease of use, flexibility, and required expertise. QuickSight might be the best starting point given your requirements, but the others are worth considering depending on your specific needs and resources.
Sources
Setting up ML-powered anomaly detection for outlier analysis - Amazon QuickSight
Amazon Lookout for Metrics features - anomaly detection service - Amazon Web Services
Transitioning off Amazon Lookout for Metrics  | AWS Machine Learning Blog