How can I collect logs from Linux instances that mount EFS file systems?

Short description

You can use an AWS Support Automation Workflow (SAW) to collect log information and other data to provide to AWS Support when troubleshooting an issue. However, you can't use SAW to collect logs for Amazon EC2 Linux instances that mount an Amazon EFS.

To collect logs from Amazon EC2 Linux instances that mount an Amazon EFS, complete the following tasks:

• Create a support case.
• Turn on stunnel logs on the client instance.
• Turn on efs-utils logs on the client instance.
• Collect logs from the Amazon EFS CSI driver.
• (Optional) Use tcpdump on the client instance to collect a packet capture.

Resolution

Create a support case

Create a support case for AWS Support to review your Amazon EFS logs. Use SSH to collect logs for the Amazon EC2 Linux instance. Include the following information in your support case:

• File System ID
• Mount Target IDs or IP addresses
• AWS Region
• EC2 Instance IDs that exhibit this behavior
• Date and time of event (include specific timestamps and time zone)
• The output from the following commands:

mount
uname -a
sudo iptables -S
lsmod

If you use efs-utils, then run the following command. Include the amazon-efs-utils.tar.gz output information in the support case:

sudo tar czf amazon-efs-utils.tar.gz /var/log/amazon/efs

If you mount over TLS, then run the following commands. Include the output information in the support case:

sudo stunnel -version
sudo openssl version
sudo openssl ciphers

Turn on stunnel logs on the client instance

1. Open /etc/amazon/efs/efs-utils.conf in a file editor. Change the stunnel_debug_enabled option for the EC2 instance from false to true.
2. Unmount, and then mount the file system to reproduce the problem that you're debugging.
3. Collect the stunnel logs. The logs are found under var/log/amazon/efs.
4. Set stunnel_debug_enabled to false.

Turn on efs-utils logs on the client instance

1. Open /etc/amazon/efs/efs-utils.conf in a file editor. Change the setting from INFO to DEBUG.
2. Unmount, and then mount the file system to reproduce the problem that you're debugging.
3. Collect the efs-utils logs. The logs are found under /var/log/amazon/efs.
4. Set efs-utils.conf to INFO.

Collect logs from aws-efs-csi-driver

1.    To find the pod that causes the error, run the following command:

kubectl describe pod $<POD_NAME>

Note: Replace <POD_NAME> with the name of your pod.

2.    Turn on the efs-utils and csi-driver debug logs for the pod that causes the error. For more information, see aws-efs-csi-driver troubleshooting guide (on the GitHub website).

3.    Run the following command to collect the debug log information:

python3 log_collector.py --driver-pod-name <driver_pod_name>

Note: Replace <driver_pod_name> with the name of your pod.

(Optional) Use tcpdump on the client instance to collect a packet capture

Run tcpdump (on the TCPDump website) to create thirty 1 GB sized files that you rotate for use. tcpdump uses the files to capture the packet data. Adjust the commands based on your requirements.

Non-TLS mounts

For non-TLS mounts, run the following command to capture plaintext network file system traffic:

sudo tcpdump -W 30 -C 1000 -s 2000 -w nfs_pcap_$(date +%FT%T).pcap -i any -z gzip -Z root port 2049

TLS mount

Run the following command to capture plaintext traffic between the network file system client and stunnel. It also captures encrypted traffic between stunnel and EFS.

sudo tcpdump -W 30 -C 1000 -s 2000 -w nfs_pcap_$(date +%FT%T).pcap -i any -z gzip -Z root