Skip to content

Global security change broke a bunch of stuff ... how to figure out what was changed

0

Ok, so last week we are starting to move to SSM for patch management and security hub but one of the 2 guys that have access didn't document what was changed and a handful of things are not working as expected.

The first is a docker container complaining about not having access to credentials, the temp fix was to allow IMDVv1 where it used to be locked down. The other one I noticed was a basic dashboard I had that queried S3 bucket size in a cloudwatch graph (using BucketSizeBytes), and that appears to have now stopped around the same time. I'm not sure why or how that change would affect, but clearly something security wise was changed and trying to rectify.

They thought it was under the main account attributes/ec2/data protection and security, so I see under the main account under IMDS was metadata version 2, so I moved that back to 1 and 2, but not sure on the other settings. So the current status is all those machines are allowing and the services are working, but looking at CloudWatch MetadataNoToken on those machines, there not using any making it even more odd. The other odd thing is older nodes that were already running are still running fine with IMDSv2 enabled. But if I make an AMI image, start a new one, it doesn't work. So is there a way to see the other options on that instance that works (metadata accessable, metadata transport) as I don't see that in the details.

Really not sure what/where to start but there maybe an easy look here or at least a point in the right direction.

3 Answers
0

All the configuration changes made to AWS resources should be logged reliably in your CloudTrail logs. If you have many accounts, are they in one or several AWS Organizations organisations, and do they have an organisation trail configured?

If the trail is available and you know which IAM roles or other form of credentials they used to make the changes, you should be able to get a full, chronological list of changes made across all your AWS accounts by querying the CloudTrail logs with Athena. In addition to filtering by identity/principal, you can write the SQL query to exclude "eventname" (=name of the API called) values starting with Describe, Get, and List to filter out the majority of events not involving modifications.

EXPERT
answered 2 years ago
0

Thanks, unfortunately no trails were on and I will surely assume that it would be from this point forward. It's just odd, as I said, even taking an image that is running right now, working with security enabled, I AMI, start a new one, and things fail.

So trying to see how to get more info on the working AMI regarding the metadata options!

Thanks again

answered 2 years ago
0

Are there many accounts where changes were, or are suspected of having been made? There's a default trail in locally in every account and region. You can't access those trails via Athena to filter and summarise them quickly, and you have to access the trail in each region separately, but all management events over the past 90 days are stored in those trails: https://docs.aws.amazon.com/awscloudtrail/latest/userguide/tutorial-event-history.html

There's also a default filter that eliminates read-only actions from the list, but unfortunately, combining multiple filters isn't possible with the event history browser. If you have some scripting skills, it's possible to export the events with the CLI for parsing and filtering from the JSON output with an external tool: https://docs.aws.amazon.com/awscloudtrail/latest/userguide/view-cloudtrail-events-cli.html

Another place to look would be AWS Config, if you happen to have it running and recording configurations for those resources. The AWS Config console makes it easy to visualise the "before" and "after" configuration states for individual changes, but it also doesn't do much to give an overview of all the changes made.

EXPERT
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.