cant connect cloudwatch log stream to mwaa public env

Question

Hello all,

I have  created public mwaa environment.
Airflow version = 2.8.1
Environment  class = mw1.small
Workers count with default values.
Region is eu-north-1

Added the following settings:

celery.sync_parallelism = 1 
core.dag_file_processor_timeout = 150
core.dagbag_import_timeout = 90 
core.min_serialized_dag_update_interval = 300
scheduler.dag_dir_list_interval = 600 
scheduler.min_file_process_interval = 300
scheduler.parsing_processes = 2 
scheduler.processor_poll_interval = 60

Added  inbound rules where types are HTTP and HTTPS and custom source where the value is the vpc security group.

Here is the policy attached to the mwaa environment:

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Effect": "Allow",
			"Action": "airflow:PublishMetrics",
			"Resource": "arn:aws:airflow:::environment/"
		},
		{
			"Effect": "Deny",
			"Action": "s3:ListAllMyBuckets",
			"Resource": [
				"arn:aws:s3:::",
				"arn:aws:s3:::/*"
			]
		},
		{
			"Effect": "Allow",
			"Action": [
				"s3:GetObject*",
				"s3:GetBucket*",
				"s3:List*"
			],
			"Resource": [
				"arn:aws:s3:::",
				"arn:aws:s3:::/*"
			]
		},
		{
			"Effect": "Allow",
			"Action": [
				"logs:CreateLogStream",
				"logs:CreateLogGroup",
				"logs:PutLogEvents",
				"logs:GetLogEvents",
				"logs:GetLogRecord",
				"logs:GetLogGroupFields",
				"logs:GetQueryResults"
			],
			"Resource": [
				"arn:aws:logs:::log-group:airflow--*"
			]
		},
		{
			"Effect": "Allow",
			"Action": [
				"logs:DescribeLogGroups"
			],
			"Resource": [
				"*"
			]
		},
		{
			"Effect": "Allow",
			"Action": "cloudwatch:PutMetricData",
			"Resource": "*"
		},
		{
			"Effect": "Allow",
			"Action": [
				"sqs:ChangeMessageVisibility",
				"sqs:DeleteMessage",
				"sqs:GetQueueAttributes",
				"sqs:GetQueueUrl",
				"sqs:ReceiveMessage",
				"sqs:SendMessage"
			],
			"Resource": "arn:aws:sqs::*:airflow-celery-*"
		},
		{
			"Effect": "Allow",
			"Action": [
				"kms:Decrypt",
				"kms:DescribeKey",
				"kms:GenerateDataKey*",
				"kms:Encrypt"
			],
			"NotResource": "arn:aws:kms:*::key/*",
			"Condition": {
				"StringLike": {
					"kms:ViaService": [
						"sqs..amazonaws.com"
					]
				}
			}
		}
	]
}

Why the DAGs won't run?

I get this error:

*** Unable to read remote logs from Cloudwatch (log_group: airflow-Airflow-Env-Task, log_stream: dag_id=test_dag/run_id=manual__2024-11-06T13_23_21.720411+00_00/task_id=print_hello_world/attempt=1.log)
*** An error occurred (ResourceNotFoundException) when calling the GetLogEvents operation: The specified log stream does not exist.

*** Could not read served logs: Request URL is missing an 'http://' or 'https://' protocol.

re:Post Agent · Answer

It appears that your MWAA environment is experiencing issues with reading remote logs from CloudWatch, which is preventing your DAGs from running properly. There are several potential reasons for this problem:

1. IAM Permissions: While your IAM policy includes permissions for CloudWatch Logs, there might be some missing or incorrect permissions. Ensure that the policy grants all necessary permissions for CloudWatch Logs operations.

2. Log Group Configuration: The error suggests that the specified log stream does not exist. This could be due to incorrect log group naming or issues with log stream creation.

3. Network Connectivity: For a public MWAA environment, ensure that the VPC has proper internet connectivity and that the security groups allow outbound traffic to CloudWatch Logs endpoints.

4. CloudWatch Agent Configuration: If you're using the CloudWatch agent, make sure it's correctly configured and running without errors.

To resolve this issue, try the following steps:

1. Double-check your IAM permissions. Ensure that the MWAA execution role has the correct permissions to access CloudWatch Logs, including creating log streams and putting log events.

2. Verify that the log group naming convention is correct. The log group should follow the format "airflow-<your-environment-name>-*".

3. Check the CloudWatch Logs console to see if the log group and log streams are being created correctly.

4. Ensure that your VPC has the necessary network configuration to allow communication with CloudWatch Logs endpoints.

5. If you're using a customer-managed KMS key for CloudWatch Logs encryption, make sure the MWAA execution role has the necessary permissions to use this key.

6. Consider increasing the logging level temporarily to get more detailed error information.

7. If the issue persists, you may need to review the MWAA environment's network configuration, including security groups and network ACLs, to ensure proper connectivity to AWS services.

If these steps don't resolve the issue, you may need to contact AWS support for further assistance, as there could be underlying configuration issues with your MWAA environment.  
**Sources**  
[Resolve the read remote logs error in Amazon MWAA | AWS re:Post](https://repost.aws/knowledge-center/mwaa-read-logs-error)  
[Resolve empty log issues in CloudWatch | AWS re:Post](https://repost.aws/knowledge-center/cloudwatch-empty-logs)  
[Troubleshoot permission errors for an instance on CloudWatch | AWS re:Post](https://repost.aws/knowledge-center/cloudwatch-instance-permission-errors)

cant connect cloudwatch log stream to mwaa public env

Relevant content