Fetching logs in Airflow UI results in a timeout

0

Recently, we can not see the logs in the Airflow UI anymore because the corresponding requests ends in a 504 (Gateway Timeout):

https://<uuid>.c3.eu-central-1.airflow.amazonaws.com/get_logs_with_metadata?dag_id=s3_example_dag&task_id=write-s3-task&execution_date=2021-02-24T08%3A58%3A58.485414%2B00%3A00&try_number=1&metadata=null

Is there a permission that is being required by this? My user that logs into the UI has Admin permissions, so this should not be the issue. We use the following Airflow configuration:

resource "aws_security_group" "airflow" {
vpc_id = var.vpc_id

egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}

ingress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}

resource "aws_mwaa_environment" "airflow" {
execution_role_arn = aws_iam_role.airflow.arn
name = "<name>"

source_bucket_arn = aws_s3_bucket.airflow.arn
dag_s3_path = aws_s3_bucket_object.dags.key
requirements_s3_path = "requirements.txt"
network_configuration {
security_group_ids = [
aws_security_group.airflow.id
]
subnet_ids = var.private_subnets
}
airflow_configuration_options = {
"secrets.backend" = "airflow.contrib.secrets.aws_systems_manager.SystemsManagerParameterStoreBackend"
}
webserver_access_mode = "PUBLIC_ONLY"
}

The logs are correctly written in CloudWatch and in the UI the tasks are also being executed. Only when we want to display the logs in the Airflow UI, it ends in a timeout. Is there any configuration that we are missing?

Thanks!

Edited by: capca5 on Feb 24, 2021 4:41 AM

Edited by: capca5 on Feb 24, 2021 4:42 AM

capca5
asked 3 years ago1105 views
2 Answers
0

Hi,

The Airflow logs are retrieved directly from CloudWatch using the MWAA Execution Role permissions. For this to function, the execution role needs permissions for, and the VPC needs connectivity to, CloudWatch logs and metrics plus the logs must be enabled in the MWAA environment at the desired level (for example INFO to see all).

Thanks!

AWS
John_J
answered 3 years ago
0

Looks like the issue was the remote secret backend. After switching to "airflow.contrib.secrets.aws_secrets_manager.SecretsManagerBackend", everything worked as expected again. In addition, it is important that the connection "aws_default" is not being overwritten. Working example:

resource "aws_secretsmanager_secret" "connection_aws" {
name = "airflow/connections/aws_default"
}

resource "aws_secretsmanager_secret_version" "connection_aws" {
secret_id = aws_secretsmanager_secret.connection_aws.id
secret_string = "aws://"
}

resource "aws_secretsmanager_secret" "connection_aws_custom" {
name = "airflow/connections/aws_custom"
}

resource "aws_secretsmanager_secret_version" "connection_aws_custom" {
secret_id = aws_secretsmanager_secret.connection_aws_custom.id
secret_string = "aws://?region_name=eu-central-1&role_arn=${aws_iam_role.task.arn}"
}

capca5
answered 3 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions