Error launching POD in EKS from MWAA

Question

We are trying to launch a POD in EKS from MWAA. Our EKS is authenticated using aws-iam-authenticator in kube_config.yaml. But MWAA shows below error in the MWAA log
kubernetes.config.config_exception.ConfigException: Invalid kube-config file. No configuration found.
MWAA Environment ARN or Name: arn:axxxxxx:environment/airflow-demo
Region: us-east-1

It looks like the DAG is unable to read the config file stored in S3. I am not sure whether its related to using the kube_config.yaml from S3 or using aws-iam-authenticator. We referred below writeup except the kubeconfig authentication part.
https://blog.beachgeek.co.uk/working-with-amazon-eks-and-amazon-managed-workflows-for-apache-airflow-v2x/

Can someone help?

Thanks
--Venky

rePost-User-8027811 · Answer

Found the root cause. Looks like below are 2 mandatory requirements for this
1) The location of the file in the DAG must be /usr/local/airflow/dags/kube_config.yaml
2) Name of the kube config yaml must be kube_config.yaml

We specified s3 folder location in the DAG and it didn't pick up. Looks like MWAA mounts the dags folder in /usr/local/airflow. After fixing this, kube_config seems be to getting loaded.

Now we are running into below error
[2023-01-26, 16:31:57 UTC] {{kubernetes_pod.py:675}} INFO - Building pod mwaa-pod-test-c7lcxfvu with labels: {'dag_id': 'kubernetes_pod_example', 'task_id': 'pod-task', 'run_id': 'manual__2023-01-26T163152.7629780000-7df477b2b', 'kubernetes_pod_operator': 'True', 'try_number': '1'}
[2023-01-26, 16:31:57 UTC] {{kube_config.py:517}} ERROR - [Errno 13] Permission denied: 'aws-iam-authenticator'
[2023-01-26, 16:31:57 UTC] {{kubernetes_pod.py:730}} ERROR - 'NoneType' object has no attribute 'metadata'

Do we need install aws-iam-authenticator in MWAA environment? If so, how to do that?

Grigorii Kushnir · Answer

You need to allow inbound traffic in your EKS cluster security group from the MWAA environment security group.

Error launching POD in EKS from MWAA

Relevant content