How do I retrieve log data from CloudWatch Logs?

Short description

There are four methods that are best practices for retrieving log data from CloudWatch Logs. You can use any one of these four, depending upon your use case:

• Use subscription filters to stream log data to another receiving source in real time
• Run a query with CloudWatch Logs Insights
• Export log data to Amazon Simple Storage Service (Amazon S3) for batch use cases
• Call GetLogEvents or FilterLogEvents in the CloudWatch API

Resolution

Use subscription filters

To immediately retrieve log data from CloudWatch Logs in real time, you can use subscription filters. For more information, see Real-time processing of log data with subscriptions and Using CloudWatch Logs subscription filters.

Filtering for log events is performed internally, which prevents CloudWatch API throttling. Amazon Kinesis Data Streams automatically retries throttled service API calls.

Run a query in CloudWatch Logs Insights

To quickly search and analyze your log data, run a query in CloudWatch Logs Insights.

Export log data to Amazon S3 (batch use cases)

To move log data from CloudWatch Logs to Amazon S3 in batch use cases, see Exporting log data to Amazon S3. Log data can take up to twelve hours to become available for export from CloudWatch Logs. For real-time analysis and processing, use subscription filters.

Important: An AWS account can have only one export task for log data in the state PENDING, PENDING_CANCEL, or RUNNING.

Call GetLogEvents or FilterLogEvents

To find log data manually, use GetLogEvents or FilterLogEvents in the CloudWatch API.

Important: This method isn’t scalable due to the transactions per second (TPS) quotas for GetLogEvents and FilterLogEvents. For current quotas, see CloudWatch Logs quotas. If you experience throttling when performing these actions, use subscription filters instead.

Related information

What is Amazon CloudWatch Logs?