Fetching logs in Airflow UI results in a timeout

Question

Recently, we can not see the logs in the Airflow UI anymore because the corresponding requests ends in a 504 (Gateway Timeout):   
  
https://.c3.eu-central-1.airflow.amazonaws.com/get_logs_with_metadata?dag_id=s3_example_dag&task_id=write-s3-task&execution_date=2021-02-24T08%3A58%3A58.485414%2B00%3A00&try_number=1&metadata=null  
  
Is there a permission that is being required by this? My user that logs into the UI has Admin permissions, so this should not be the issue. We use the following Airflow configuration:   
  
resource "aws_security_group" "airflow" {  
  vpc_id = var.vpc_id  
  
  egress {  
    from_port   = 0  
    to_port     = 0  
    protocol    = "-1"  
    cidr_blocks = \["0.0.0.0/0"]  
  }  
  
  ingress {  
    from_port   = 0  
    to_port     = 0  
    protocol    = "-1"  
    cidr_blocks = \["0.0.0.0/0"]  
  }  
}  
  
resource "aws_mwaa_environment" "airflow" {  
  execution_role_arn = aws_iam_role.airflow.arn  
  name               = ""  
  
  source_bucket_arn    = aws_s3_bucket.airflow.arn  
  dag_s3_path          = aws_s3_bucket_object.dags.key  
  requirements_s3_path = "requirements.txt"  
  network_configuration {  
    security_group_ids = \[  
      aws_security_group.airflow.id  
    ]  
    subnet_ids         = var.private_subnets  
  }  
  airflow_configuration_options = {  
    "secrets.backend" = "airflow.contrib.secrets.aws_systems_manager.SystemsManagerParameterStoreBackend"  
  }  
  webserver_access_mode = "PUBLIC_ONLY"  
}  
  
The logs are correctly written in CloudWatch and in the UI the tasks are also being executed. Only when we want to display the logs in the Airflow UI, it ends in a timeout. Is there any configuration that we are missing?   
  
Thanks!  
  
Edited by: capca5 on Feb 24, 2021 4:41 AM  
  
Edited by: capca5 on Feb 24, 2021 4:42 AM

Answer

Hi,  
  
The Airflow logs are retrieved directly from CloudWatch using the MWAA Execution Role permissions.  For this to function, the execution role needs permissions for, and the VPC needs connectivity to, CloudWatch logs and metrics plus the logs must be enabled in the MWAA environment at the desired level (for example INFO to see all).  
  
Thanks!

Answer

Looks like the issue was the remote secret backend. After switching to "airflow.contrib.secrets.aws_secrets_manager.SecretsManagerBackend", everything worked as expected again. In addition, it is important that the connection "aws_default" is not being overwritten. Working example:   
  
resource "aws_secretsmanager_secret" "connection_aws" {  
  name = "airflow/connections/aws_default"  
}  
  
resource "aws_secretsmanager_secret_version" "connection_aws" {  
  secret_id     = aws_secretsmanager_secret.connection_aws.id  
  secret_string = "aws://"  
}  
  
resource "aws_secretsmanager_secret" "connection_aws_custom" {  
  name = "airflow/connections/aws_custom"  
}  
  
resource "aws_secretsmanager_secret_version" "connection_aws_custom" {  
  secret_id     = aws_secretsmanager_secret.connection_aws_custom.id  
  secret_string = "aws://?region_name=eu-central-1&role_arn=${aws_iam_role.task.arn}"  
}

Fetching logs in Airflow UI results in a timeout

Relevant content