Kubernetes pod Operator with mwaa eks s3

0

Hi I am trying to run this in mwaa I am able to pull image form ecr repo and spin eks pod but I am getting error when its trying to run file inside that pod like cmds=["python","s3://mwaa-bucket/dags/test.py"]

I tried it with s3 URL and even giving s3 airflow dag folder path didn't work

Here is snippet for reference podRun = KubernetesPodOperator( namespace="mwaa", #image="ubuntu:18.04", image = "xxxxxxxxxx.dkr.ecr.us-east-2.amazonaws.com/new_repo:latest", cmds=["python","/usr/local/airflow/dags/test.py"], arguments=["-c", "ls"], labels={"foo": "bar"}, name="mwaa-pod-test", task_id="pod-task", get_logs=True, dag=dag, is_delete_operator_pod=False, config_file=kube_config_path, in_cluster=False, cluster_context='aws' )

2 Answers
1

The S3 bucket used by MWAA is not sync'd to the image used by Kubernetes. As such, your pod must get its code there itself.

See https://docs.aws.amazon.com/mwaa/latest/userguide/mwaa-eks-example.html

AWS
John_J
answered a year ago
0

In which section should I make a change for this in https://docs.aws.amazon.com/mwaa/latest/userguide/mwaa-eks-example.html let me know thanks for a response

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions