Kubernetes pod Operator with mwaa eks s3

0

Hi I am trying to run this in mwaa I am able to pull image form ecr repo and spin eks pod but I am getting error when its trying to run file inside that pod like cmds=["python","s3://mwaa-bucket/dags/test.py"]

I tried it with s3 URL and even giving s3 airflow dag folder path didn't work

Here is snippet for reference podRun = KubernetesPodOperator( namespace="mwaa", #image="ubuntu:18.04", image = "xxxxxxxxxx.dkr.ecr.us-east-2.amazonaws.com/new_repo:latest", cmds=["python","/usr/local/airflow/dags/test.py"], arguments=["-c", "ls"], labels={"foo": "bar"}, name="mwaa-pod-test", task_id="pod-task", get_logs=True, dag=dag, is_delete_operator_pod=False, config_file=kube_config_path, in_cluster=False, cluster_context='aws' )

2 Respuestas
1

The S3 bucket used by MWAA is not sync'd to the image used by Kubernetes. As such, your pod must get its code there itself.

See https://docs.aws.amazon.com/mwaa/latest/userguide/mwaa-eks-example.html

AWS
John_J
respondido hace un año
0

In which section should I make a change for this in https://docs.aws.amazon.com/mwaa/latest/userguide/mwaa-eks-example.html let me know thanks for a response

respondido hace un año

No has iniciado sesión. Iniciar sesión para publicar una respuesta.

Una buena respuesta responde claramente a la pregunta, proporciona comentarios constructivos y fomenta el crecimiento profesional en la persona que hace la pregunta.

Pautas para responder preguntas