Tools for IAM roles and policies regression testing

Question

My daily job is to update IAM roles (about 4k roles) and customer managed IAM policies (about 10k policies). The IAM policies definitions are stored in terraform code. To avoid mistakes in roles and policies updates, I wish there would be something like regression testing framework for IAM roles/policies documents which I can run 50-100 times a day while developing new version of policy documents in terraform. Main issue is the lack of confidence in new IAM policies versions and potential customer impact in case of IAM permissions regression.

Has someone encountered similar problem and how did you address it?

Accepted Answer

Hi Denys,

Thank you for the clarification. I can dive into solutions, but I need to understand why you are doing what you are doing first.

Are those 30 accounts part of an AWS Organization [1] with an OU structure set up? If yes, how are you providing guardrails? For example, take a look at SCPs [2] and how these can help in setting guardrails. I mentioned SCPs because this can safe guard someone creating a policy which is too open, especially since you are (semi)manually dealing with some many accounts.

Since you are using Terraform, and for simplicity sake I assume you have an AWS Organization set up for those accounts, I would advise looking into AWS Control Tower in combination with Terraform for account provisioning and customization [3][4][5].

The core of your questions is check the validity of a new and/or changed policy. I would strongly advise to build in automation for testing and deployment of, in this case, policies. There is a lot that can be said about this, which is nicely captured in our DevOps Guidance [6].

Also, who is creating those policies? Can you set an automatic pipeline for them to perform some checking with the AWS CLI [7].  I must say that the results of the aforementioned command can differ from your AWS environment.

The above in combination with a high level of automation should shed light on the correct approach.

[1] https://docs.aws.amazon.com/organizations/latest/userguide/orgs_introduction.html

[2] https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_scps.html

[3] https://www.youtube.com/watch?v=8Ot5wn7kxI0

[4] https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-and-manage-aws-control-tower-controls-by-using-terraform.html

[5] https://docs.aws.amazon.com/controltower/latest/userguide/what-is-control-tower.html

[6] https://docs.aws.amazon.com/wellarchitected/latest/devops-guidance/devops-guidance.html

[7] https://awscli.amazonaws.com/v2/documentation/api/latest/reference/iam/simulate-custom-policy.html

Cheers,

Andrei

Answer

Hi Denys, 
With AWS Config, you can get detailed configurations for AWS resources including IAM users, roles and policies.

1. Enable AWS Config with one-click setup
2. include globally recorded resource types such as IAM users, policies
3. AWS Config Advanced query provide SQL query service for AWS services
4. Also, Resource Inventory provide the history of resource changes.

example)
```
SELECT
  resourceId,
  resourceName,
  resourceType,
  tags,
  configuration.attachedManagedPolicies,
  availabilityZone
WHERE
  resourceType LIKE 'AWS::IAM::Policy'
```

You can check the detailed information in the below link.
- https://docs.aws.amazon.com/config/latest/developerguide/querying-AWS-resources.html

Answer

Hi Andrei,

Thank you for the response! I work with >30 related aws accounts, which happened to have large number of customer managed policies. Overall, roles and policies in different accounts follow a pattern, e.g. an account has all resource within dedicated vpc, an s3 bucket is accessible from within vpc, there is a fixed list of kms keys per account, etc. However, these patterns evolves over time. Sometimes, it is extra security requirement, sometimes additional access to a new service has to be provided, sometimes there are deviations between different accounts and so on. Updating policy documents at this scale is a painstaking process with potentially undesirable fallout. I have an opportunity to preview changes in a kind of testing environment (yet another account), therefore the question.  For now, I write terraform, deploy it in the test environment, go to the console, assume role, check, run aws cli commands locally, to increase the confidence in the new version of policy documents (this is kind of semi-automated regression testing step). I realize how easy it is to introduce e.g. wrong resource arn into the policy document simply by using wrong variable in the code.

I was hoping that more experienced engineers reading the forum have already found a way to streamline the process and may want to share their approach or simply share their vision on the problem.

Answer

Amazing suggestions, Andrei! AWS Organization and Control Tower are perhaps less useful in my case, but I'm very thankful for the references to DevOps and simulate-custom-policy pages. I wasn't familiar with them. I quickly glanced through the pages and I like what I've read so far. I'll try to find inspiration there. Thanks a lot!

Answer

Hi Denys,

Let's first start by understanding why you need to update that many roles and policies. What is the rationale behind it? Perhaps you could also share a bit more information on the setup of you AWS account(s).

Kind regards,
Andrei Scorus

Tools for IAM roles and policies regression testing

Relevant content