How do I get SSM agent logs for Fargate tasks that have Amazon ECS Exec activated?

Short description

Prerequisite:

Complete the requirements to use ECS Exec.

To get SSM agent logs for Fargate tasks that have ECS Exec activated, create an Amazon Elastic File System (Amazon EFS) file system. Then, mount the Amazon EFS file system on the Fargate container. Finally, mount the same file system on an Amazon Elastic Compute Cloud (Amazon EC2) instance to get the SSM Agent logs.

Important: Before you mount the EFS file system, you must launch an Amazon EC2 instance. Your Amazon EFS file system, Amazon ECS cluster, and Fargate tasks must all be in the same Amazon Virtual Private Cloud (Amazon VPC).

Resolution

Note: The following resolution is only for Fargate tasks that have ECS Exec activated. Use the resolution steps only to debug. To be sure that you don't override your logs, launch the task as a standalone task. Or, keep your desiredCount of tasks to "1" in your Amazon ECS service. You can also use the following resolution for scenarios where you must check non stderr/stdout logs from the containers.

Create your Amazon EFS file system and mount it on a Fargate container

To create your Amazon EFS file system and mount it on a Fargate container in a task or service, complete the following steps:

1. Create your Amazon EFS file system.
2. Note the Amazon EFS ID and security group ID.
3. Edit your file system security group rules to allow inbound connections on port 2049 from the security group that's associated with your Fargate task.
4. Update your Amazon ECS security group to allow outbound connections on port 2049 to your file system's security group.
5. Open the Amazon ECS console.
6. In the navigation pane, choose Task definitions, and then choose your task definition.
7. Choose Create new revision.
8. Under Storage, for Volumes, choose Add volume. Then, enter the following information
   For Volume name, enter a name for your volume.
   For Volume type, enter choose EFS.
   For File system ID, enter the ID for your file system.
9. For Container mount points, choose Add mount point. Then, enter the following information:
   For Container, choose your container.
   For Source volume, choose the source volume.
   For Container path, enter /var/log/amazon.
10. Choose Create.

Note: Before you mount the Amazon EFS file system to an Amazon EC2 instance, the task must be in a running state.

Mount the Amazon EFS file system on an Amazon EC2 instance and get the SSM Agent logs

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshoot AWS CLI errors. Also, make sure that you're using the most recent AWS CLI version.

Complete the following steps:

1. Mount your file system on an EC2 instance.

2. To get the log data, run the following AWS CLI command:

   sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport fs-01b0bxxxxxxxx.efs.ap-southeast-1.amazonaws.com:/ /efs

   Example output:

   # df -h
   Filesystem                                          Size  Used Avail Use% Mounted on
   fs-01b0bxxxxxxxx.efs.us-west-2.amazonaws.com:/      8.0E     0  8.0E   0% /efs

The following is an example of logs stored at path /var/log/amazon/ssm/amazon-ssm-agent.log in the Fargate container:

[root@ip-172-31-32-32 efs]# cd ssm/
[root@ip-172-31-32-32 ssm]# ls
amazon-ssm-agent.log  audits
[root@ip-172-31-32-32 ssm]# cat amazon-ssm-agent.log | tail -n 10
2022-10-20 11:50:34 INFO [ssm-agent-worker] [MessageService] [MessageHandler] ended idempotency deletion thread
2022-10-20 11:50:37 INFO [ssm-agent-worker] [MessageService] [MGSInteractor] send failed reply thread started
2022-10-20 11:50:37 INFO [ssm-agent-worker] [MessageService] [MGSInteractor] send failed reply thread done
2022-10-20 11:55:37 INFO [ssm-agent-worker] [MessageService] [MGSInteractor] send failed reply thread started
2022-10-20 11:55:37 INFO [ssm-agent-worker] [MessageService] [MGSInteractor] send failed reply thread done
2022-10-20 12:00:34 INFO [ssm-agent-worker] [MessageService] [MessageHandler] started idempotency deletion thread
2022-10-20 12:00:34 WARN [ssm-agent-worker] [MessageService] [MessageHandler] [Idempotency] encountered error open /var/lib/amazon/ssm/170b15cacf5846ed836bcd7903cbee48-2531612879/idempotency: no such file or directory while listing replies in /var/lib/amazon/ssm/170b15cacf5846ed836bcd7903cbee48-2531612879/idempotency
2022-10-20 12:00:34 INFO [ssm-agent-worker] [MessageService] [MessageHandler] ended idempotency deletion thread
2022-10-20 12:00:37 INFO [ssm-agent-worker] [MessageService] [MGSInteractor] send failed reply thread started
2022-10-20 12:00:37 INFO [ssm-agent-worker] [MessageService] [MGSInteractor] send failed reply thread done
[root@ip-172-31-32-32 ssm]#