EC2 Instance Loses Docker Deployment & Disappears from LB Target Groups – Ephemeral Storage or AMI Issue?

Question

I’m relatively new to AWS and working with an ecosystem that was set up by another engineer using scripting tools. I’m facing an issue in our development/test environment where an EC2 instance running a Docker container to serve static files over HTTPS suddenly loses its deployed code and is removed from the load balancer target groups.

**Environment & Setup:**
* 	Compute: EC2 instance launched from an AMI built via scripting.
* 	Storage: The instance uses an EBS volume with the root device /dev/xvda (15GiB), and the volume appears healthy and is in active use.
* 	Networking: Configured within a VPC with appropriate security groups, and we’re using a load balancer linked to target groups.
* 	Additional Services: Auto Scaling is enabled, and other components include EBS volumes, IAM for access, and the setup likely involves other AWS services such as VPC configurations.

**Issue Details:**
* 	I connect to the EC2 instance using Session Manager, install Docker and Git, and then run a Docker container that serves static files over HTTPS.
* 	Initially, everything works fine – the load balancer health checks pass, and external clients can connect to the endpoint.
* 	After a random period (usually a couple of days), external clients can no longer reach the endpoint.
* 	When I check the instance via Session Manager, the deployed code (and the Docker container) is gone.
* 	Additionally, I’ve observed that the load balancer target groups no longer list this instance (or any instances) after the issue occurs.
* 	We don't have many clients at all (less than 100 total) since this is a dev environment. We also notice our server seems to slow down a bit before the deployment fails  –  this could be unrelated but noteworthy.

**Questions & Concerns:**
1. 	Ephemeral Storage vs. EBS: Could this be related to ephemeral storage? I’m confused because if the root device is mounted as EBS, shouldn’t the data persist across instance lifecycles?
2. 	AMI Implications: Is it possible that the issue is connected to how the AMI was created or is being used in conjunction with Auto Scaling?
3. 	Instance Replacement: Could an automated process be replacing the instance, hence the loss of my manual deployments?
4. 	Best Practices: What are the best practices for ensuring that application code persists on an instance deployed via scripts, especially when making post-launch modifications?

I can reinstall the code without issue, but it’s important that the application persists for our deployment. Any insights, diagnostic suggestions, or pointers to relevant AWS documentation would be greatly appreciated.

Thank you for your help!

re:Post Agent · Answer

Based on the information provided, it appears that your EC2 instance is likely being replaced by the Auto Scaling process, which explains the loss of your manually deployed code and the instance's disappearance from the load balancer target groups. This is a common scenario in environments using Auto Scaling, especially when instances are configured to be replaced periodically or in response to certain conditions.

To address your specific questions and concerns:

1. Ephemeral Storage vs. EBS: The issue is not related to ephemeral storage. Your EBS volume is persistent, but when a new instance is launched by Auto Scaling, it starts with a fresh copy of the AMI, which doesn't include your manual deployments.

2. AMI Implications: The AMI is likely the base image used by Auto Scaling to launch new instances. Any changes you make after the instance is launched (like installing Docker and deploying your code) are not part of this base image.

3. Instance Replacement: Yes, it's highly probable that an automated process (Auto Scaling) is replacing the instance. This explains why your manual deployments are lost and why the instance disappears from the target groups.

4. Best Practices: To ensure application code persists across instance replacements:

a. Update your AMI: Include Docker, Git, and your application code in the AMI itself.
   b. Use user data scripts: Configure the instance to pull and run your Docker container on launch.
   c. Implement a CI/CD pipeline: Automate the deployment process to ensure new instances are properly configured.
   d. Use ECS or EKS: These services can manage container deployments more effectively.
   e. Utilize persistent storage: Store important data on separate EBS volumes or use EFS for shared storage across instances.

The slowdown you observe before the deployment fails could be related to the instance being marked unhealthy by Auto Scaling, triggering a replacement.

To resolve this issue:

1. Review your Auto Scaling configuration, including launch templates or launch configurations.
2. Update your AMI or use user data scripts to automate the deployment process.
3. Implement proper health checks to ensure instances are not replaced unnecessarily.
4. Consider using AWS developer tools like CodeDeploy to manage application deployments in an Auto Scaling environment.

By implementing these practices, you can create a more robust and persistent deployment that survives instance replacements in your Auto Scaling group.  
**Sources**  
[EC2 Instance Disappeared | AWS re:Post](https://repost.aws/questions/QUWkbJeNdBSrCW9Ie23fLptw/ec2-instance-disappeared)  
[EC2 Instance has disappeared | AWS re:Post](https://repost.aws/questions/QUAdW3NiAHT6GqcrFvFOy8iA/ec2-instance-has-disappeared)  
[Beanstalk platform updates remove the instances from their load balancer target groups | AWS re:Post](https://repost.aws/questions/QU_hjC2mTRQ4a0NEmFNxTWsA/beanstalk-platform-updates-remove-the-instances-from-their-load-balancer-target-groups)  
[ECS: Tasks deployed on a terminating EC2 instance | AWS re:Post](https://repost.aws/questions/QUr5BbeHWiQTKWK5Z-0Ns_Og/ecs-tasks-deployed-on-a-terminating-ec2-instance)

EC2 Instance Loses Docker Deployment & Disappears from LB Target Groups – Ephemeral Storage or AMI Issue?

Add your answer

Relevant content