How do I resolve "Forbidden" errors when calling certain Apache Airflow REST API endpoints in Amazon MWAA?

Short description

This Forbidden error occurs because different Apache Airflow REST API endpoints require different role-based access control (RBAC) permissions, and the Airflow role mapped to your IAM policy determines which endpoints you can access. The /eventLogs endpoint requires the Audit Logs.can_read permission, which is restricted to the Admin role by default in Apache Airflow version 2.9.2 and later. If your IAM policy uses a wildcard ("Resource": "*") or specifies a non-Admin role in the resource ARN, the session is mapped to a role that lacks this permission.

For more information about Amazon MWAA access policies, see Accessing an Amazon MWAA environment.

Resolution

Understand why some endpoints work and others don't

Apache Airflow uses RBAC to control access to its REST API endpoints. Each endpoint requires specific permissions, and each default Airflow role (Admin, Op, User, Viewer, Public) has a different set of permissions.

The /dags/{dag_id}/dagRuns/{dag_run_id} endpoint requires the DAGs.can_read and DAG Runs.can_read permissions, which are available to the Viewer role and above. The /eventLogs endpoint requires the Audit Logs.can_read permission, which is available only to the Admin role by default. This permission restriction was tightened in Apache Airflow version 2.9.2 as part of a security fix (CVE-2024-26280) that addressed incorrect default permissions for audit log access.

Because Amazon MWAA version 3.0 runs Apache Airflow v3, this restriction applies.

Understand how Amazon MWAA maps IAM policies to Airflow roles

Amazon MWAA determines your Airflow RBAC role from the Resource ARN in your IAM policy. The Resource ARN follows this format:

arn:aws:airflow:REGION:ACCOUNT_ID:role/ENVIRONMENT_NAME/AIRFLOW_ROLE

Note: Replace REGION with your AWS Region (for example, us-east-1), ACCOUNT_ID with your 12-digit AWS account ID, ENVIRONMENT_NAME with your Amazon MWAA environment name, and AIRFLOW_ROLE with one of the following values: Admin, Op, User, Viewer, or Public.

If your IAM policy uses "Resource": "*" instead of a specific role ARN, Amazon MWAA defaults to the most restrictive role. This results in insufficient permissions for endpoints that require elevated access.

Identify which access method you are using

Amazon MWAA supports two methods for accessing the Apache Airflow REST API. Identify which method you are using, then follow the corresponding steps.

Method 1: Direct webserver access using a web session token

If you are calling the REST API using the webserver VPCE URL directly (for example, https://YOUR-ENVIRONMENT-ID.YOUR-REGION.airflow.YOUR-REGION.on.aws/api/v2/eventLogs), you are using the web session token method. This method uses the airflow:CreateWebLoginToken IAM action.

Method 2: InvokeRestApi using AWS SDK or CLI

If you are calling the REST API through the AWS SDK (boto3) using client.invoke_rest_api(), you are using the InvokeRestApi method. This method uses the airflow:InvokeRestApi IAM action.

Update your IAM policy to specify the Admin role

1. Open the IAM console.
2. In the navigation pane, choose Policies.
3. Find and choose the IAM policy attached to the user, group, or role that accesses your Amazon MWAA environment.
4. Choose Edit.
5. Update the policy to specify the Admin Airflow role in the Resource ARN for both actions.

The following example policy grants Admin-level access for both the InvokeRestApi and CreateWebLoginToken actions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "airflow:InvokeRestApi",
                "airflow:CreateWebLoginToken"
            ],
            "Resource": "arn:aws:airflow:REGION:ACCOUNT_ID:role/ENVIRONMENT_NAME/Admin"
        }
    ]
}

Note: Replace REGION with your AWS Region, ACCOUNT_ID with your 12-digit AWS account ID, and ENVIRONMENT_NAME with your Amazon MWAA environment name.

6. Choose Review policy, then choose Save changes.
7. Wait two to three minutes for the IAM policy changes to propagate.

Verify the fix

If using the web session token method:

1. Generate a new web login token. Existing sessions use the old role mapping, so you must create a new session after updating the IAM policy.

2. Run the following Python script to test the /eventLogs endpoint:

import boto3
import requests

mwaa_client = boto3.client("mwaa")

# Step 1: Get a web login token
# Note: Replace YOUR_ENVIRONMENT_NAME with your Amazon MWAA environment name
token_response = mwaa_client.create_web_login_token(
    Name="YOUR_ENVIRONMENT_NAME"
)

web_server_hostname = token_response["WebServerHostname"]
web_token = token_response["WebToken"]

# Step 2: Exchange the token for a session
# Note: For Amazon MWAA version 3.0, the login path is /pluginsv2/aws_mwaa/login
login_url = f"https://{web_server_hostname}/pluginsv2/aws_mwaa/login"
login_response = requests.get(
    login_url,
    params={"token": web_token},
    allow_redirects=False
)

# Step 3: Extract the JWT token
jwt_token = login_response.cookies["_token"]

# Step 4: Call the eventLogs endpoint
headers = {
    "Authorization": f"Bearer {jwt_token}",
    "Content-Type": "application/json"
}

api_url = f"https://{web_server_hostname}/api/v2/eventLogs"
response = requests.get(api_url, headers=headers)
print(response.status_code)
print(response.json())

If using the InvokeRestApi method:

Run the following Python script to test the /eventLogs endpoint:

import boto3

client = boto3.client("mwaa")

# Note: Replace YOUR_ENVIRONMENT_NAME with your Amazon MWAA environment name
response = client.invoke_rest_api(
    Name="YOUR_ENVIRONMENT_NAME",
    Path="/eventLogs",
    Method="GET",
)

print(response["RestApiResponse"])

A successful response returns a 200 status code with a JSON body containing the event log entries.

Additional considerations

Private webserver environments: If your Amazon MWAA environment uses a private webserver, the InvokeRestApi action must be called from within the VPC. Use the aws:SourceVpc condition key in your IAM policy to enforce this restriction.

Least-privilege access: If granting the Admin role is too broad for your use case, create a custom Airflow role that includes only the Audit Logs.can_read permission. To create custom roles, sign in to the Apache Airflow UI as an Admin user, navigate to Security, then choose List Roles. For more information about custom roles, see Tutorial: Restricting an Amazon MWAA user's access to a subset of DAGs.

Amazon MWAA version 3.0 login path changes: If you upgraded from Amazon MWAA version 2.x to 3.0, the login path changed from /aws_maa/login to /pluginsv2/aws_mwaa/login, and the REST API base path changed from /api/v1 to /api/v2. Update your scripts accordingly.

Related information

• Using the Apache Airflow REST API
• Access Control with FAB auth manager
• Audit Logs in Airflow
• Best practices for least privilege configuration in Amazon MWAA
• Introducing simplified interaction with the Airflow REST API in Amazon MWAA