Fetching logs in Airflow UI results in a timeout

Recently, we can not see the logs in the Airflow UI anymore because the corresponding requests ends in a 504 (Gateway Timeout):

https://<uuid>.c3.eu-central-1.airflow.amazonaws.com/get_logs_with_metadata?dag_id=s3_example_dag&task_id=write-s3-task&execution_date=2021-02-24T08%3A58%3A58.485414%2B00%3A00&try_number=1&metadata=null

Is there a permission that is being required by this? My user that logs into the UI has Admin permissions, so this should not be the issue. We use the following Airflow configuration:

resource "aws_security_group" "airflow" {
vpc_id = var.vpc_id

egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}

ingress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}

resource "aws_mwaa_environment" "airflow" {
execution_role_arn = aws_iam_role.airflow.arn
name = "<name>"

source_bucket_arn = aws_s3_bucket.airflow.arn
dag_s3_path = aws_s3_bucket_object.dags.key
requirements_s3_path = "requirements.txt"
network_configuration {
security_group_ids = [
aws_security_group.airflow.id
]
subnet_ids = var.private_subnets
}
airflow_configuration_options = {
"secrets.backend" = "airflow.contrib.secrets.aws_systems_manager.SystemsManagerParameterStoreBackend"
}
webserver_access_mode = "PUBLIC_ONLY"
}

The logs are correctly written in CloudWatch and in the UI the tasks are also being executed. Only when we want to display the logs in the Airflow UI, it ends in a timeout. Is there any configuration that we are missing?

Thanks!

Edited by: capca5 on Feb 24, 2021 4:41 AM

Edited by: capca5 on Feb 24, 2021 4:42 AM