Logging with CloudWatch vs. ElasticSearch/Kibana

Comparison

Cloudwatch Logs offers highly durable centralized storage for logs. It also allows customers to track things like the number of errors appearing in the logs and create alerts. It also supports simple querying (http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/FilterAndPatternSyntax.html) Customers do not need to worry about cluster sizing, storage, scaling etc - there are not servers to manage.

Elasticsearch with Kibana provide a lot more functionality in terms of searching and visualizing your logs. While our managed Elasticsearch service removes a lot of the undifferentiated heavy lifting customers still need to put some thought and monitoring to ensure they have the right capacity in place and that they backup Elasticsearch data to a more durable location like S3.

Cloudwatch Logs AND Elasticsearch

• Cloudwatch Logs -> Elasticsearch

Customers can get best of both worlds by streaming CloudWatch Logs into Elasticsearch http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_ES_Stream.html

• Logstash -> Cloudwatch Logs

Since they are already pushing their data into Logstash this plugin can send the data to Cloudwatch Logs: https://github.com/awslabs/logstash-output-cloudwatchlogs

• Cloudwatch Agent -> Cloudwatch Logs

However the Cloudwatch Logs agent is probably the easiest way to get logs into CloudWatch logs: http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_GettingStarted.html

hope this helps

PS if they decide not to use Cloudwatch Logs here are the options:

Elasticsearch (without Cloudwatch Logs)

• Logstash -> Elasticsearch

Since they are already using Logstash the easiest way to get started would be to use this connector to send data to Amazon Elasticsearch https://github.com/awslabs/logstash-output-amazon_es

• Kinesis Firehose -> Elasticsearch

Of course there is also the option to replace logstash with Kinesis Firehose which removes the operational burden of managing Logstash EC2 instances: https://aws.amazon.com/kinesis/firehose/firehose-to-elasticsearch-service/