How can I get data to help troubleshoot IAM permission access denied or unauthorized errors?

5 minute read

I receive an "access denied" or "unauthorized" error when I access an AWS resource. I want data to help troubleshoot these AWS Identity and Access Management (IAM) API call failure errors.

Short description

Use Amazon Athena queries or the AWS Command Line Interface (AWS CLI) to get error logs for IAM API call failures. Then, follow the instructions to troubleshoot access denied or unauthorized operation errors with an IAM policy.

Resolution

Note: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent version of the AWS CLI.

Use Athena queries to troubleshoot IAM API call failures by searching CloudTrail logs

Note: Before you begin, you must have a trail created to log to an Amazon Simple Storage Service (Amazon S3) bucket. This is because Athena uses events recorded in AWS CloudTrail log files that are delivered to an Amazon S3 bucket for that trail.

1. Follow the steps in the create the Athena table section of How do I automatically create tables in Athena to search through AWS CloudTrail logs?

Note: Athena tables that are created automatically are in the same AWS Region as your Amazon S3 bucket.

2. Open the Athena console, and then choose the plus sign "+" to create a new query.

3. Enter the following example query, and then choose Run.

In this example query, the time format uses ISO 8601 basic format with the Z variable for UTC.

Note: Replace your-arn with the IAM Amazon Resource Names (ARN) for your resources and your-table with your table name.

SELECT from_iso8601_timestamp(eventTime) AS "Time", useridentity.arn AS "Identity ARN", eventID AS "Event ID",
         eventsource AS "Service", eventname AS "Action", errorCode AS "Error", errorMessage AS "Message"
FROM your-table
WHERE from_iso8601_timestamp(eventtime) >= from_iso8601_timestamp('2019-10-29T06:40:00Z')
        AND from_iso8601_timestamp(eventtime) < from_iso8601_timestamp('2019-10-29T06:55:00Z')
        AND userIdentity.arn = 'your-arn'
        AND eventType = 'AwsApiCall'
        AND errorCode is not null
        AND (lower(errorCode) LIKE '%accessdenied%' OR lower(errorCode) LIKE '%unauthorized%')
ORDER BY eventTime desc

4. This example table output lists permission errors for the identity ARN:

| Time                        | Event ID                             | Service                  | Action       | Error        | Message                                                                                                              |
|-----------------------------|--------------------------------------|--------------------------|--------------|--------------|----------------------------------------------------------------------------------------------------------------------|
| 2019-10-29 06:52:45.000 UTC | 0406f0c1-47a8-4f71-8a94-18267b84042a | cloudtrail.amazonaws.com | LookupEvents | AccessDenied | User: arn:aws:iam::account:user/username is not authorized to perform: cloudtrail:LookupEvents with an explicit deny in an identity-based policy |
| 2019-10-29 06:41:48.000 UTC | 14e5e77c-f682-45e1-8c88-12d15af293dd | cloudtrail.amazonaws.com | LookupEvents | AccessDenied | User: arn:aws:iam::account:user/username is not authorized to perform: cloudtrail:LookupEvents because no identity-based policy allows the cloudtrail:LookupEvents action |

Note: CloudTrail event outputs can take up to 15 minutes to deliver results.

5. Optionally, get errors for all users by removing this line from the example query:

AND userIdentity.arn = 'your-arn'

6. Optionally, get all errors from a selected time period by removing this line from the example query:

AND (lower(errorCode) LIKE '%accessdenied%' OR lower(errorCode) LIKE '%unauthorized%')

Use the AWS CLI to troubleshoot IAM permission API call failures

Note: This AWS CLI script requires the jq command line JSON processor. For the tutorial and download instructions, see JSON output format. For distributions that use the yum package, run the following command:

$ sudo yum install jq

1. Run the following AWS CLI command:

Note: Replace your-arn with the IAM ARNs for your resources.

( echo "Time,Identity ARN,Event ID,Service,Action,Error,Message";
  aws cloudtrail lookup-events --start-time "2019-10-29T06:40:00Z" --end-time "2019-10-29T06:55:00Z" --query "Events[*].CloudTrailEvent" --output text \
    | jq -r ". | select(.userIdentity.arn == \"your-arn\" and .eventType == \"AwsApiCall\" and .errorCode != null
    and (.errorCode | ascii_downcase | (contains(\"accessdenied\") or contains(\"unauthorized\"))))
    | [.eventTime, .userIdentity.arn, .eventID, .eventSource, .eventName, .errorCode, .errorMessage] | @csv"
) | column -t -s'",'

Note: The rate of lookup requests to CloudTrail is limited to two requests per second, per account, per Region. If this limit is exceeded, a throttling error occurs.

2. This example table output lists permission errors for the identity ARN from the specified time period.

Note: You can look up events that occurred in a Region from the last 90 days.

Time                  Event ID                              Service                   Action        Error         Message
2019-10-29T06:52:45Z  0406f0c1-47a8-4f71-8a94-18267b84042a  cloudtrail.amazonaws.com  LookupEvents  AccessDenied  User: arn:aws:iam::account:user/username is not authorized to perform: cloudtrail:LookupEvents with an explicit deny in an identity-based policy
2019-10-29T06:41:48Z  14e5e77c-f682-45e1-8c88-12d15af293dd  cloudtrail.amazonaws.com  LookupEvents  AccessDenied  User: arn:aws:iam::account:user/username is not authorized to perform: cloudtrail:LookupEvents because no identity-based policy allows the cloudtrail:LookupEvents action

3. (Optional) get errors for all users by removing this line:

.userIdentity.arn == \"your-arn\" and

4. (Optional) get all errors from the selected time period by removing this line:

and (.errorCode | ascii_downcase | (contains(\"accessdenied\") or contains(\"unauthorized\")))

Troubleshoot unauthorized errors

Athena and the previous AWS CLI example outputs are relevant to CloudTrail LookupEvents API calls.

IAM policies that deny access because it contains a Deny statement include a specific phrase in the error message for explicit and implicit denies. IAM explicit deny errors contain the phrase "with an explicit deny in a <type> policy". IAM implicit deny errors contain the phrase "because no <type> policy allows the <action> action".

The cloudtrail:LookupEvents with an explicit deny output indicates that an associated IAM policy is incorrect.

An explicit deny can occur from any of these policy types. For example, identity-based policies, resource-based policies, permissions boundaries, organizations SCPs, and session policies. Explicit deny statements always override allow statements. The explicit deny exists in the IAM users identity-based policy.

The cloudtrail:LookupEvents because no identity-based policy allows output indicates that the identity-based policy doesn't allow this API action resulting in an implicit deny. The identity-based policy lacks an explicit allow statement for the cloudtrail:LookupEvents API action.

Policy types evaluated by AWS to establish access are:

For additional information about how IAM policies are evaluated and managed, see Policy evaluation logic and Managing IAM policies.