How do I troubleshoot the error "nfs: server 127.0.0.1 not responding" when I mount my Amazon EFS file system?

Short description

You might see the "server not responding" error for the following reasons:

• The NFS client can't connect to the Amazon EFS server.
• A reboot or shutdown of the instance caused a network disconnection.
• Resources are over-utilized.
• Network performance issues are causing errors.
• There are issues with the specific kernel version.
• You're using an incorrect mount configuration.

Complete the solution method that best fits your use case.

Resolution

NFS client can't connect to the Amazon EFS server

If the NFS client can't connect to the Amazon EFS server, then check the network configuration for the following issues: incorrect security group settings, DNS resolution problems, or VPC configurations.

To check the connection, run the following command:

telnet efs-mount-target-IP-address 2049

If the connection is successful, then the output shows a connection to the specified port (EFS mount target IP address):

ec2-user@ip-172-31-8-105 ~
telnet 172.31.11.149 2049 
Trying 172.31.11.149... 
Connected to 172.31.11.149. 
Escape character is '^'.

If the connection fails, then complete the following checks:

• Check that the security group associated with your EFS file system allows inbound traffic on port 2049 from the client's security group. For more information, see Creating security groups.
• Check that the DNS can resolve the EFS mount target's IP address. For more information, see Mounting on Amazon EC2 with a DNS name.

A reboot or shutdown of the instance occurred

To check if a reboot or shutdown occurred, review the StatusCheckFailed metric in Amazon CloudWatch for recent instance stops or reboots.

Utilization is high

High CPU or memory usage can lead to performance issues. To check the CPU and memory usage, use a tool such as atop. For more information, see A guide to atop command in Linux on the Digital Ocean website. Then, review the CloudWatch metric CPUUtilization for your EC2 instance.

To resolve this issue, update your instance to a larger instance type.

Network performance issues

CloudWatch metrics

Poor network performance can result in EFS mount errors. To monitor your network performance, complete the following steps:

1. Open the CloudWatch console.
2. Select your EC2 instance.
3. In the Metrics section, under EC2, check the following metrics:
   Network packets in/out: High variance in packet rate can show network instability.
   Network bytes in/out: High variance in byte rate can show network instability.
   Network errors: An increase in network errors can signify issues with the network interface. To resolve a high number of network errors, change the instance type to one with increased network performance.
   Network throughput: Sustained high throughput shows that you require a higher bandwidth instance.

Other monitoring tools

You can also use third party tools such as sar, nload, and iftop on the Die Net website.

Identify and mitigate micro-bursting

Micro-bursting can cause sudden performance degradation because of brief spikes in network traffic that can lead to packet drops and potential errors. To identify and mitigate micro-bursting, complete the following steps:

1. Open the CloudWatch console.
2. Select your EC2 instance.
3. Choose Monitoring, and then review the following metrics:
   Network In/Out: Monitor the volume of incoming and outgoing network traffic.
   Network Packets In/Out: Check for sudden spikes in the number of packets that are sent or received over short intervals.
   CPU Credit Usage and CPU Credit Balance: Check whether the instance is experiencing CPU limitations that correlate with network performance issues.
   Note: Sudden increases in network traffic or packet rates can show micro-bursting. It's a best practice to monitor these metrics over time to identify patterns of brief spikes.

Use the data that you gathered to look for brief, sudden increases in network traffic or packet counts. If micro-bursting is confirmed, then complete the following steps:

• Adjust your workload distribution to smooth out network traffic and prevent sudden spikes.
• Upgrade to a higher-performance instance or use an Elastic Network Adapter (ENA) activated instance to handle network demands.

For more information, see How can I identify if my Amazon EBS volume is micro-bursting and then prevent this from happening?

Troubleshoot kernel version issues

Some kernel versions have issues that can cause EFS mount failures.

To check the kernel version, run the following command:

uname -r

The output looks similar to the following:

6.1.97-104.177.amzn2023.x86_64

Make sure that you're using kernel version 4.3 or newer.

Incorrect mount configuration

If you use incorrect mount options, then the mount fails.

To validate your mount options, check the mount command used. When you mount the file system, make sure that you include the noresvport option:

sudo mount -t nfs -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport <mount-target-ip>:/ <mount-point>

For more information about mount options, see Recommended NFS mount settings.

If you're using the correct mount configuration but it fails, then collect your log information and contact AWS Support.