How do I troubleshoot Apache Airflow UI access issues in Amazon MWAA?

Short description

When you try to access the Apache Airflow UI in Amazon MWAA, you might receive Oops, something went wrong or 403 Forbidden errors. These errors occur when your AWS Identity and Access Management (IAM) role doesn't have the necessary permissions.

You might also receive server errors and 5xx errors. These errors occur because of DNS issues, Amazon Virtual Private Cloud (Amazon VPC) connection issues, or incompatibilities with the environment's requirements.

Amazon MWAA provides Public and Private network access mode for the Apache Airflow UI. If you receive connection timeout errors in Private mode, then make sure that your intermediary mechanism is correctly configured in your Amazon VPC.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Troubleshooting the "Oops, something went wrong" error or 403 Forbidden errors

To troubleshoot the Oops, something went wrong error or 403 Forbidden errors, check the IAM access policy permissions for the IAM role. Edit your IAM access role policy to include the airflow:CreateWebLoginToken permission:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "airflow:CreateWebLoginToken",
            "Resource": "arn:aws:airflow:example-region:example-account-id:role/example-mwaa-environment/example-role";
        }
    ]
}

Note: Replace example-region with your AWS Region, example-account-id with your account ID, and example-role with Admin, Op, User, Viewer, or Public. For UI access, your IAM access policy must align with a predefined role-based access control (RBAC) role that Apache Airflow supports. For more information, see Apache Airflow UI access policy: AmazonMWAAWebServerAccess.

Troubleshoot 403 Forbidden errors because of expired web login tokens

Web login tokens have a 60-second validity period. For private web server access, use a mechanism such as a Bastion Host or AWS Client VPN.

To generate a web login token before you access the UI, run the create-web-login-token command:

token=$(aws mwaa create-web-login-token --name example-mwaa-environment)
WEB_TOKEN=$(echo $token | jq --raw-output '.WebToken')
echo $WEB_TOKEN

Note: Replace example-mwaa-environment with your Amazon MWAA environment name.

To access the Apache Airflow UI behind an Application Load Balancer, use the following format for your URL:

https://example-alb-dns/aws_mwaa/aws-console-sso?login=true#example-web-token

Note: Replace example-alb-dns with your Application Load Balancer DNS and example-web-token with your web token.

For Amazon VPC access, use browser extensions, such as LiveHosts or Internal Redirector, to redirect the Airflow endpoint to the Application Load Balancer DNS.

Troubleshoot server errors and 5xx errors

Before you begin, run the Amazon MWAA verify script from AWS support tools to check your environment's health. For more information, see aws-support-tools on the GitHub website. Then, check your execution role permissions, DNS settings and network configuration, and your resource contention.

Check your execution role permissions

To review and edit your execution role policy, see View and update an execution role policy.

Check your DNS settings and network configuration

Make sure that your DNS settings and network configuration allow access to the web server and metadatabase.

To troubleshoot your DNS settings and network configuration, take the following actions:

• For DNS resolution, run a dig command to make sure that the Airflow UI's domain name resolves correctly.
• To test your connectivity to the Airflow UI endpoint from your Amazon VPC, use telnet or curl.
• To configure your network access control lists (network ACLs) and route tables, follow the Amazon MWAA security guidelines for Amazon VPC.
• To allow traffic on ports 443 and 5432, modify your security groups. If you use an Application Load Balancer, then modify your security groups from the Application Load Balancer.
• To meet the Amazon MWAA networking requirements, use an AWS CloudFormation template to create an Amazon VPC network with internet access.

Check your resource contention

Scheduler, web server, and worker components that have high CPU and memory utilization might cause UI access issues. Also, incorrect DAG syntax or compatibility issues might cause resource contention. If these issues occur, then you receive an Oops, something bad has happened error or 5xx errors when you access the Airflow UI. To check for resource contention, review your CPUUtilization and MemoryUtilization Amazon CloudWatch metrics for high CPU and memory usage.

To determine if one of your DAGs is causing access issues, remove all Python files from your environment's DAGs location in Amazon Simple Storage Service (Amazon S3). Then, add them back one at a time and monitor for changes in UI access or performance to identify the affected DAG.

For more information, see How do I resolve common issues with my scheduler in Amazon MWAA?

Check your Python package version requirements

Incompatible Python package versions might cause Airflow UI access issues. To troubleshoot this issue, check for incompatible dependencies or missing constraints. Also, check your Airflow logs in CloudWatch for the Airflow scheduler, worker, and web server. Check for errors that are related to DAG processing, requirements installation, or plugin runtime.

For more information, see How do I install libraries in my Amazon MWAA environment?

Related information

I see a 5xx error accessing the web server