AWS MWAA Environment error INCORRECT_CONFIGURATION using existing VPC (not created by MWAA)

Hi team,

I saw several questions which relate to my question:

• MWAA stuck in a loop while Creating Environment
• AWS MWAA Environment error "INCORRECT_Configuration"

But, I still can't resolve the issue. I did verify the MWAA environment with aws-support-tools for MWAA.

Here is the detail:

please ignore on the Testing connectivity to the following service endpoints from MWAA enis... part since the AWS MWAA creation is failed so there is no active connectivity.

(venv) [ec2-user@xxxx verify_env]$ python ./verify_env.py --envname airflow-reproduce-error-2
please send support the following information
If a case is not opened you may open one here https://console.aws.amazon.com/support/home#/case/create
Please make sure to NOT include any personally identifiable information in the case

AirflowConfigurationOptions :  {}
AirflowVersion :  2.5.1
Arn :  arn:aws:airflow:ap-southeast-1:xxxxx:environment/airflow-reproduce-error-2
CreatedAt :  2023-05-09 08:30:03+00:00
DagS3Path :  dags
EnvironmentClass :  mw1.small
ExecutionRoleArn :  arn:aws:iam::xxxxx:role/service-role/AmazonMWAA-airflow-reproduce-error-2-gtEO4G
LastUpdate :  {'CreatedAt': datetime.datetime(2023, 5, 9, 8, 30, 3, tzinfo=tzlocal()), 'Error': {'ErrorCode': 'INCORRECT_CONFIGURATION', 'ErrorMessage': 'You may need to check the execution role permissions policy for your environment, and that each of the VPC networking components required by the environment are configured to allow traffic. Troubleshooting: https://docs.aws.amazon.com/mwaa/latest/userguide/troubleshooting.html'}, 'Status': 'FAILED'}
LoggingConfiguration :  {'DagProcessingLogs': {'Enabled': False, 'LogLevel': 'WARNING'}, 'SchedulerLogs': {'Enabled': True, 'LogLevel': 'WARNING'}, 'TaskLogs': {'Enabled': True, 'LogLevel': 'INFO'}, 'WebserverLogs': {'Enabled': True, 'LogLevel': 'WARNING'}, 'WorkerLogs': {'Enabled': True, 'LogLevel': 'WARNING'}}
MaxWorkers :  10
MinWorkers :  1
Name :  airflow-reproduce-error-2
NetworkConfiguration :  {'SecurityGroupIds': ['sg-0d69257297ef7bf77'], 'SubnetIds': ['subnet-061f5b0d526d19b32', 'subnet-052af55a94725413d']}
RequirementsS3ObjectVersion :  null
RequirementsS3Path :  requirements.txt
Schedulers :  2
ServiceRoleArn :  arn:aws:iam::xxxxx:role/aws-service-role/airflow.amazonaws.com/AWSServiceRoleForAmazonMWAA
SourceBucketArn :  arn:aws:s3:::xxxxx-airflow-stg
Status :  CREATE_FAILED
Tags :  {}
WebserverAccessMode :  PUBLIC_ONLY
WeeklyMaintenanceWindowStart :  THU:23:30
VPC:  vpc-06a7b7e73e27af6d6 

### Checking the IAM execution role arn:aws:iam::xxxxx:role/service-role/AmazonMWAA-airflow-reproduce-error-2-gtEO4G using iam policy simulation
Using AWS CMK
Action: airflow:PublishMetrics is allowed on resource arn:aws:airflow:ap-southeast-1:xxxxx:environment/airflow-reproduce-error-2 ✅
Action: s3:ListAllMyBuckets is blocked successfully on resource arn:aws:s3:::xxxxx-airflow-stg ✅
Action: s3:ListAllMyBuckets is blocked successfully on resource arn:aws:s3:::xxxxx-airflow-stg/ ✅
Action: s3:GetObject* is allowed on resource arn:aws:s3:::xxxxx-airflow-stg ✅
Action: s3:GetObject* is allowed on resource arn:aws:s3:::xxxxx-airflow-stg/ ✅
Action: s3:GetBucket* is allowed on resource arn:aws:s3:::xxxxx-airflow-stg ✅
Action: s3:GetBucket* is allowed on resource arn:aws:s3:::xxxxx-airflow-stg/ ✅
Action: s3:List* is allowed on resource arn:aws:s3:::xxxxx-airflow-stg ✅
Action: s3:List* is allowed on resource arn:aws:s3:::xxxxx-airflow-stg/ ✅
Action: logs:CreateLogStream is allowed on resource arn:aws:logs:ap-southeast-1:xxxxx:log-group:airflow-airflow-reproduce-error-2-* ✅
Action: logs:CreateLogGroup is allowed on resource arn:aws:logs:ap-southeast-1:xxxxx:log-group:airflow-airflow-reproduce-error-2-* ✅
Action: logs:PutLogEvents is allowed on resource arn:aws:logs:ap-southeast-1:xxxxx:log-group:airflow-airflow-reproduce-error-2-* ✅
Action: logs:GetLogEvents is allowed on resource arn:aws:logs:ap-southeast-1:xxxxx:log-group:airflow-airflow-reproduce-error-2-* ✅
Action: logs:GetLogGroupFields is allowed on resource arn:aws:logs:ap-southeast-1:xxxxx:log-group:airflow-airflow-reproduce-error-2-* ✅
Action: logs:DescribeLogGroups is allowed on resource * ✅
Action: cloudwatch:PutMetricData is allowed on resource * ✅
Action: sqs:ChangeMessageVisibility is allowed on resource arn:aws:sqs:ap-southeast-1:*:airflow-celery-* ✅
Action: sqs:DeleteMessage is allowed on resource arn:aws:sqs:ap-southeast-1:*:airflow-celery-* ✅
Action: sqs:GetQueueAttributes is allowed on resource arn:aws:sqs:ap-southeast-1:*:airflow-celery-* ✅
Action: sqs:GetQueueUrl is allowed on resource arn:aws:sqs:ap-southeast-1:*:airflow-celery-* ✅
Action: sqs:ReceiveMessage is allowed on resource arn:aws:sqs:ap-southeast-1:*:airflow-celery-* ✅
Action: sqs:SendMessage is allowed on resource arn:aws:sqs:ap-southeast-1:*:airflow-celery-* ✅
Action: kms:Decrypt is allowed on resource arn:aws:kms:*:111122223333:key/* ✅
Action: kms:DescribeKey is allowed on resource arn:aws:kms:*:111122223333:key/* ✅
Action: kms:Encrypt is allowed on resource arn:aws:kms:*:111122223333:key/* ✅
Action: kms:GenerateDataKey* is allowed on resource arn:aws:kms:*:111122223333:key/* ✅
If the policy is denied you can investigate more at 
https://policysim.aws.amazon.com/home/index.jsp?#roles/AmazonMWAA-airflow-reproduce-error-2-gtEO4G

These simulations are based off of the sample policies here 
https://docs.aws.amazon.com/mwaa/latest/userguide/mwaa-create-role.html#mwaa-create-role-json

### Checking if log groups were created successfully...

number of log groups match suggesting they've been created successfully ✅
### Trying to verify nACLs on subnets...
nacl: acl-0b2023fba14350376 allows port 5432 on egress ✅
nacl: acl-0b2023fba14350376 allows port 5432 on ingress ✅

### Trying to verify if route tables are valid...

### Verifying 'block public access' is enabled on the s3 bucket or account...
Checking if public access is blocked at the bucket level
Checking if public access is blocked at the account level
The account level access block config is not set
s3 bucket, arn:aws:s3:::xxxxx-airflow-stg, or account blocks public access ✅

### Trying to verifying ingress on security groups...
ingress for security groups have at least 1 rule to allow itself ✅ 

### Testing connectivity to the following service endpoints from MWAA enis...
no enis found for MWAA, exiting test for  sqs.ap-southeast-1.amazonaws.com
please try accessing the airflow UI and then try running this script again
no enis found for MWAA, exiting test for  api.ecr.ap-southeast-1.amazonaws.com
please try accessing the airflow UI and then try running this script again
no enis found for MWAA, exiting test for  monitoring.ap-southeast-1.amazonaws.com
please try accessing the airflow UI and then try running this script again
no enis found for MWAA, exiting test for  kms.ap-southeast-1.amazonaws.com
please try accessing the airflow UI and then try running this script again
no enis found for MWAA, exiting test for  s3.ap-southeast-1.amazonaws.com
please try accessing the airflow UI and then try running this script again
no enis found for MWAA, exiting test for  env.airflow.ap-southeast-1.amazonaws.com
please try accessing the airflow UI and then try running this script again
no enis found for MWAA, exiting test for  env.airflow.ap-southeast-1.amazonaws.com
please try accessing the airflow UI and then try running this script again
no enis found for MWAA, exiting test for  ops.airflow.ap-southeast-1.amazonaws.com
please try accessing the airflow UI and then try running this script again
no enis found for MWAA, exiting test for  api.airflow.ap-southeast-1.amazonaws.com
please try accessing the airflow UI and then try running this script again
no enis found for MWAA, exiting test for  logs.ap-southeast-1.amazonaws.com
please try accessing the airflow UI and then try running this script again

### Checking CloudWatch logs for any errors less than 1 hour old
Found the following failing logs in cloudwatch: 
Log group:  airflow-airflow-reproduce-error-2-Scheduler
Log group:  airflow-airflow-reproduce-error-2-Task
Log group:  airflow-airflow-reproduce-error-2-WebServer
Log group:  airflow-airflow-reproduce-error-2-Worker

When I created a new MWAA env with new VPC, the MWAA env successfully created. I guess the issue comes from the VPC because I'm using the existing VPC and don't want the MWAA created new one for me. But I'm struggling how to investigate it.

Any suggestions on how to check further about it?

Is there any end to end tutorial how to setup new MWAA with existing VPC?

Thanks!