I want to set up the ELK stack for a production site. Could you please recommend the appropriate EC2 server configuration?

When considering a cost-effective solution for running the ELK stack (Elasticsearch, Logstash, and Kibana) in production, I'd recommend evaluating whether self-hosting on EC2 is actually the most cost-effective approach.

Self-hosting the ELK stack on EC2 comes with significant operational overhead. Installation is rarely straightforward, requiring careful configuration of data ingestion limits, retention plans, and optimal hardware sizing. You'll need to manage this across all your environments, handle resiliency issues, and maintain the system over time.

Instead of self-hosting, consider using Amazon OpenSearch Service (formerly Amazon Elasticsearch Service), which provides the benefits of the ELK stack without the operational overhead. This managed service handles infrastructure management, implementation, and ongoing maintenance.

If you still prefer to self-host on EC2, here are some cost-optimization recommendations:

1. Instance Selection:

• Consider AWS Graviton-based instances which deliver up to 40% better price performance than comparable instances
• EC2 Flex instances are good for applications that don't fully utilize compute resources, offering up to 19% better price performance

2. Purchasing Options:

• Use a mix of Reserved Instances for predictable workloads to lock in lower hourly costs
• Consider Spot Instances for non-critical, stateless components to save up to 90% compared to On-Demand prices
• Use On-Demand instances only for unpredictable workloads

3. Auto Scaling:

• Implement Auto Scaling Groups to dynamically adjust capacity based on demand
• Use AWS Compute Optimizer to analyze performance and identify right-sizing opportunities

4. Monitoring:

• Leverage AWS Cost Explorer and Trusted Advisor to identify underutilized resources and cost-saving opportunities

Remember that the appropriate configuration will depend on your specific workload requirements, including the volume of logs, retention period, and query patterns.
Sources
Optimizing Costs for EC2 in High-Traffic Scenarios? | AWS re:Post
The benefits of the ELK stack without the operational overhead
Amazon EC2 Cost and Capacity Optimization | Amazon Web Services