CloudWatch Log Correlation from EKS

Yes, it is possible to correlate logs created by EKS pods with an X-Ray trace ID, including logs from containers running in those pods. This correlation allows you to associate the container/pod logs with the same trace that includes the API Gateway and Lambda calls. Here's how you can achieve this:

1. Propagate the trace context: Ensure that the X-Ray trace ID is propagated from the Lambda function to the EKS pod. This can be done by passing the trace ID as part of the request or environment variables when launching the pod.

2. Instrument your application: Use the X-Ray SDK in your containerized application to continue the trace. This will allow your application to be aware of the trace context.

3. Log the trace ID: In your application code running in the EKS pod, make sure to include the X-Ray trace ID in your log statements. This is crucial for correlating the logs later.

4. Configure log collection: Use the CloudWatch agent or Fluent Bit to collect logs from your EKS pods. These tools can be configured to send the container logs to CloudWatch Logs.

5. Use the correct log format: When logging, use a structured format (like JSON) that includes the trace ID. This makes it easier to query and correlate logs later.

6. Query logs using CloudWatch Logs Insights: You can use CloudWatch Logs Insights to query logs across different log groups, filtering by the trace ID to correlate logs from API Gateway, Lambda, and your EKS pods.

By following these steps, you can tag/annotate the container/pod logs with the X-Ray trace ID, allowing you to associate them with the same trace that includes the API Gateway and Lambda calls. This approach provides a complete picture of the request flow, including the logs from your containerized applications running in EKS.

Remember that while this method allows for correlation, the EKS pod logs won't automatically appear in the X-Ray console alongside the API Gateway and Lambda logs. You'll need to use CloudWatch Logs Insights or a custom solution to view all correlated logs together.
Sources
Community | Easily Monitor Containerized Applications with Amazon CloudWatch Container Insights
Designing and implementing logging and monitoring with CloudWatch FAQ - AWS Prescriptive Guidance
Logs and metrics supported by Amazon CloudWatch Application Insights - Amazon CloudWatch
Community | Easily Monitor Containerized Applications with Amazon CloudWatch Container Insights