Skip to content

t4g.small instance stops responding

0

I have a t4g.small on-demand instance running in the Mumbai region. In the last couple months it has gone into a state where it stops responding. Status checks start failing and I cannot ssh into it. The first time around a new instance id was generated after reboot. Yesterday when it happened the same instance id was used. So I could pull out system logs.

It turns out the network connection failed, it has a message like this - Sep 10 17:18:59 ip-.ap-south-1.compute.internal systemd-networkd[1413]: ens5: Could not set DHCPv4 address: Connection timed out Sep 10 17:19:06 ip-.ap-south-1.compute.internal systemd-networkd[1413]: ens5: Failed

I can attach the full set of relevant logs if needed.

Is this a known issue with this instance type? Or is there any other explanation for this?

Thanks

asked a year ago273 views

3 Answers
0

This issue with your t4g.small instance becoming unresponsive is not specifically related to the instance type, but rather appears to be a network connectivity problem that can affect EC2 instances in general.

Based on the logs you shared, your instance is experiencing a DHCP timeout issue where it's unable to obtain or renew its IP address configuration. When this happens, the network interface (ens5) fails, causing the instance to become unreachable via SSH, and status checks begin to fail.

There are several potential causes and troubleshooting steps you can take:

  1. Check system logs: When you regain access to the instance, review the complete system logs to identify any patterns or additional error messages that might provide more context.

  2. Reboot the instance: As you've discovered, rebooting the instance often resolves the issue temporarily. This is because it forces the instance to request a new DHCP lease.

  3. Stop and start the instance: Rather than just rebooting, stopping and starting the instance might be more effective as it migrates your instance to new underlying hardware, which could resolve infrastructure-related issues.

  4. Monitor resource utilization: Check if the instance is running out of resources (CPU, memory) which might be affecting network services.

  5. Check for scheduled tasks: Verify if there are any cron jobs or scheduled tasks that might be running around the time of failure that could be affecting network connectivity.

  6. Instance recovery: Consider enabling automatic recovery for your instance, which can help when system status checks fail.

If the problem persists after trying these steps, you might want to consider:

  • Creating a new instance from a known good AMI
  • Evaluating if the t4g.small instance type provides enough resources for your workload
  • Implementing a monitoring solution that can detect when your instance becomes unresponsive

This doesn't appear to be a known issue specific to t4g.small instances, but rather a network configuration or infrastructure issue that can affect EC2 instances in general.
Sources
Troubleshoot Amazon EC2 Linux instances with failed status checks - Amazon Elastic Compute Cloud
EC2 instance randomly fails 1/2 status checks at 4:30 AM with no user activity — SSH and APIs become unresponsive | AWS re:Post

answered a year ago

EXPERT

reviewed a year ago

0

You said "Status checks start failing" - is it instance or system status checks? If system status check - then you should stop the instance and then start it. It will use a different host. (Note, if you restart, it will use the same host and the problem will persist. If instance status check - then something is wrong with your instance. Are you using your own AMI or AWS provided ones? If you are using your own AMI, it's possible that it's corrupted - as in you have data like IP address, etc that's causing the AMI to fail.

EXPERT

answered a year ago

0

I want to clarify, I did stop the instance and then start it. That retained the same instance id but gave me a new IP address.

I did not pay attention to the exact status check but 2 out of 3 status checks failed. Is there a way for me to go back and check now, I don't have Cloudwatch logs enabled.

I am using AWS AMI - al2023-ami-2023.7.20250527.1-kernel-6.1-arm64.

The CPU monitoring was showing it had reached around ~71% around the time the issue happened. The logs from my python API server don't have anything out of the ordinary. But the Network In bytes had spiked to 5.81M around the time the issue started happening. Can that be an issue?

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.