- Newest
- Most votes
- Most comments
Hello,
Did you assign the necessary permissions directly to the EC2 IAM role or are you using a user profile to run AWS CLI commands?
If you haven’t already, you should attach the required permissions directly to the IAM role associated with the EC2 instance. So if you are using Auto Scaling Groups you can attach it to the role in the Launch Template.
Alternatively, I suggest considering ECS, if your goal with ECR is to run a container. ECS can handle containerization and scaling for you much more seamlessly.
Hope this helps! 😊
The issue you're experiencing is likely related to how the EC2 instance is obtaining its credentials during the user data script execution versus when you SSH into the instance later.
When your user data script runs during instance startup, it seems that the EC2 instance hasn't fully acquired its IAM role credentials yet. This is why you're seeing the "User: arn:aws:iam::123456789012:user/foo" in the aws sts get-caller-identity output, which appears to be using your personal IAM user credentials instead of the instance profile.
On the other hand, when you SSH into the instance later, the instance has had time to fully acquire its IAM role credentials, which is why the aws ecr get-login-password command works correctly at that point.
To resolve this issue, you can try the following approaches:
-
Add a delay in your user data script before executing AWS CLI commands. This gives the instance more time to acquire its IAM role credentials. For example, you could add a sleep command for 10-15 seconds before running your AWS CLI commands.
-
Use the EC2 instance metadata service to explicitly retrieve the temporary credentials for the IAM role. You can do this by querying the instance metadata endpoint for the security credentials. Then use these credentials to set the appropriate environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN) before running your AWS CLI commands.
-
If possible, consider moving the ECR login process to a separate script that runs after the instance has fully started, rather than in the user data script. You could use a systemd service or a cron job to run this script a short time after the instance has booted.
By implementing one of these approaches, you should be able to ensure that your EC2 instance is using its assigned IAM role credentials when executing AWS CLI commands during startup.
Sources
Run commands when you launch an EC2 instance with user data input - Amazon Elastic Compute Cloud
Before You Begin - AMS Advanced Application Developer's Guide
Before You Begin - AMS Advanced Onboarding Guide
Access instance metadata for an EC2 instance - Amazon Elastic Compute Cloud
I was doing something stupid. I had to export env vars from a file and didn't care much what kind of variables were exported. It turns out there were aws credentials in env vars. it took me a while to find out but it wasn't by luck. As I was looking for the cause I found a helpful advice in finding my error.
The trick was to call a aws cli command with --debug flag. For example,
aws s3 --debug
Then if you carefully parse through the logs there will appear a section related to finding and obtaining aws credentials for executing the aws cli command. There I found that credentials were being picked up from environment variables.
I hope this helps.
Relevant content
- asked 3 years ago
- Accepted Answerasked a year ago
AWS OFFICIALUpdated a year ago- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 5 months ago
Thank you for the comment. I assume the necessary permissions are attached to the IAM role because it works when I connect to the instance using ssh. In my experience ECS is not desirable for applications which requires persistent connection. With Auto Scaling Group when a scale in event occurs the termination of the instance can be delayed using lifecycle hooks make sure all connections are drained. But I haven't found a good way to achieve that with ECS.
There are multiple ways to assign permissions in AWS, but since you noticed differences with the instance profile, it’s better to use the IAM role attached to the EC2 instance to grant the necessary permissions. Can you confirm that the permissions are attached to your EC2 role? As for ECS, it can integrate with an Auto Scaling Group. If you already have the desired behavior with ASG, you can link it to your ECS cluster. This way, you'll get the benefits of ECS for managing containers while keeping the scaling behavior you're comfortable with.