AMI Customizations disappear mysteriously after boot

0

Over the past few weeks, I've been seeing a very strange issue. At first I thought I must have been missing a step in my AMI creation process, but last night it happened again. Here's what we did and experienced:

Used a RHEL-7 AMI as a base, and we installed our own codebase and libraries on there. From that we created an AMI. As we've discovered more dependencies and updated the codebase, we've updated the AMI to include the new stuff.

Last night then, I booted up an instance from the AMI. We ssh'd in (via AWS SSM Session Manager) and checked and everything was working. Randomly about 40 minutes later it stopped pinging into our pingdom dashboard.

In trying to investigate: we couldn't start an AWS SSM session. Instead we had to ssh in via the old jump host.
On the instance intself -- the ssm-agent is not even present, not running. The custom config files (crontab and scoutdaemon) are no longer even on the system! The instance just looks like some plain RHEL from a couple of months ago, not the one we booted last night.

Has anyone seen this happen before? What's the underlying cause / what am I doing wrong?

This reduces our confidence in the stability of this

seemant
asked 4 years ago245 views
2 Answers
0

Hello seemant,

Could you please share the specific instance IDs, the region in question, the times and dates related to each change (at least approximate if you do not know the exact times, and also please include the time zone), the AMI IDs with the steps that you have taken all chronologically described? We do not have visibility into the instances internally, nor we can see into the AMIs when they are not public, but we might be able to help you to pinpoint the issue if we have more specific information. Further, if you have a Premium Support plan, you can create a new support case to have this all reviewed within the SLA.

Thank you for your understanding.

Regards,
awstomas

AWS
answered 4 years ago
0

Thank you. I'll close this post for now as I'm no longer generating custom AMIs at the moment

seemant
answered 4 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions