Tools for IAM roles and policies regression testing


My daily job is to update IAM roles (about 4k roles) and customer managed IAM policies (about 10k policies). The IAM policies definitions are stored in terraform code. To avoid mistakes in roles and policies updates, I wish there would be something like regression testing framework for IAM roles/policies documents which I can run 50-100 times a day while developing new version of policy documents in terraform. Main issue is the lack of confidence in new IAM policies versions and potential customer impact in case of IAM permissions regression.

Has someone encountered similar problem and how did you address it?

asked 7 months ago324 views
5 Answers
Accepted Answer

Hi Denys,

Thank you for the clarification. I can dive into solutions, but I need to understand why you are doing what you are doing first.

Are those 30 accounts part of an AWS Organization [1] with an OU structure set up? If yes, how are you providing guardrails? For example, take a look at SCPs [2] and how these can help in setting guardrails. I mentioned SCPs because this can safe guard someone creating a policy which is too open, especially since you are (semi)manually dealing with some many accounts.

Since you are using Terraform, and for simplicity sake I assume you have an AWS Organization set up for those accounts, I would advise looking into AWS Control Tower in combination with Terraform for account provisioning and customization [3][4][5].

The core of your questions is check the validity of a new and/or changed policy. I would strongly advise to build in automation for testing and deployment of, in this case, policies. There is a lot that can be said about this, which is nicely captured in our DevOps Guidance [6].

Also, who is creating those policies? Can you set an automatic pipeline for them to perform some checking with the AWS CLI [7]. I must say that the results of the aforementioned command can differ from your AWS environment.

The above in combination with a high level of automation should shed light on the correct approach.










answered 6 months ago
profile picture
reviewed 6 days ago
profile picture
reviewed 2 months ago

Hi Denys, With AWS Config, you can get detailed configurations for AWS resources including IAM users, roles and policies.

  1. Enable AWS Config with one-click setup
  2. include globally recorded resource types such as IAM users, policies
  3. AWS Config Advanced query provide SQL query service for AWS services
  4. Also, Resource Inventory provide the history of resource changes.


  resourceType LIKE 'AWS::IAM::Policy'

You can check the detailed information in the below link.

profile pictureAWS
answered 6 months ago
  • Thank you @taekyung! I haven't used AWS config before. Currently I use aws cli to download/parse policy documents. AWS Config looks to be good alternative. Unfortunately, I'm afraid it won't help to kind of 'dry run' development version of the policy documents as one typically does with 'classical' unit tests.

    I realize my question is more of open-ended discussion of pros-/cons- of different alternatives. I'll perhaps wait a day or two and accept your answer, if there will be no further ideas.

  • Simply, If you want to test your IAM permissions for dry-run. There is a 'dry-run' option. You can see the detail in the below link.

    OPTIONS --dry-run | --no-dry-run (boolean) Checks whether you have the required permissions for the action, without actually making the request, and provides an error response. If you have the required permissions, the error response is DryRun- Operation . Otherwise, it is UnauthorizedOperation .

  • Thanks @taekyung! I'll keep --dry-run flag in mind.


Amazing suggestions, Andrei! AWS Organization and Control Tower are perhaps less useful in my case, but I'm very thankful for the references to DevOps and simulate-custom-policy pages. I wasn't familiar with them. I quickly glanced through the pages and I like what I've read so far. I'll try to find inspiration there. Thanks a lot!

answered 6 months ago

Hi Denys,

Let's first start by understanding why you need to update that many roles and policies. What is the rationale behind it? Perhaps you could also share a bit more information on the setup of you AWS account(s).

Kind regards, Andrei Scorus

answered 6 months ago

Hi Andrei,

Thank you for the response! I work with >30 related aws accounts, which happened to have large number of customer managed policies. Overall, roles and policies in different accounts follow a pattern, e.g. an account has all resource within dedicated vpc, an s3 bucket is accessible from within vpc, there is a fixed list of kms keys per account, etc. However, these patterns evolves over time. Sometimes, it is extra security requirement, sometimes additional access to a new service has to be provided, sometimes there are deviations between different accounts and so on. Updating policy documents at this scale is a painstaking process with potentially undesirable fallout. I have an opportunity to preview changes in a kind of testing environment (yet another account), therefore the question. For now, I write terraform, deploy it in the test environment, go to the console, assume role, check, run aws cli commands locally, to increase the confidence in the new version of policy documents (this is kind of semi-automated regression testing step). I realize how easy it is to introduce e.g. wrong resource arn into the policy document simply by using wrong variable in the code.

I was hoping that more experienced engineers reading the forum have already found a way to streamline the process and may want to share their approach or simply share their vision on the problem.

answered 6 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions