Error launching POD in EKS from MWAA

Question

We are trying to launch a POD in EKS from MWAA. Our EKS is authenticated using aws-iam-authenticator in kube_config.yaml. But MWAA shows below error in the MWAA log
kubernetes.config.config_exception.ConfigException: Invalid kube-config file. No configuration found.
MWAA Environment ARN or Name: arn:axxxxxx:environment/airflow-demo
Region: us-east-1

It looks like the DAG is unable to read the config file stored in S3. I am not sure whether its related to using the kube_config.yaml from S3 or using aws-iam-authenticator. We referred below writeup except the kubeconfig authentication part.
https://blog.beachgeek.co.uk/working-with-amazon-eks-and-amazon-managed-workflows-for-apache-airflow-v2x/

Can someone help?

Thanks
--Venky

Answer

Found the root cause. Looks like below are 2 mandatory requirements for this
1) The location of the file in the DAG must be /usr/local/airflow/dags/kube_config.yaml
2) Name of the kube config yaml must be kube_config.yaml

We specified s3 folder location in the DAG and it didn't pick up. Looks like MWAA mounts the dags folder in /usr/local/airflow. After fixing this, kube_config seems be to getting loaded.

Now we are running into below error
[2023-01-26, 16:31:57 UTC] {{kubernetes_pod.py:675}} INFO - Building pod mwaa-pod-test-c7lcxfvu with labels: {'dag_id': 'kubernetes_pod_example', 'task_id': 'pod-task', 'run_id': 'manual__2023-01-26T163152.7629780000-7df477b2b', 'kubernetes_pod_operator': 'True', 'try_number': '1'}
[2023-01-26, 16:31:57 UTC] {{kube_config.py:517}} ERROR - [Errno 13] Permission denied: 'aws-iam-authenticator'
[2023-01-26, 16:31:57 UTC] {{kubernetes_pod.py:730}} ERROR - 'NoneType' object has no attribute 'metadata'

Do we need install aws-iam-authenticator in MWAA environment? If so, how to do that?

Answer

You need to allow inbound traffic in your EKS cluster security group from the MWAA environment security group.

Error launching POD in EKS from MWAA

Relevant content